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EXECUTIVE  SUMMARY 


Global  Positioning  System  (GPS)  technology  is  becoming  more  widely  used  and  may 
eventually  replace  existing  precision  navigation  systems  (e.g.,  Instrument  Landing  System). 
To  a  large  degree,  current  CPS  receiver  designs  are  based  on  previous  Long-Range 
Navigation  (LORAN)  receiver  designs.  A  problem  with  this  approach  is  that  current 
LORAN  receiver  designs  have  been  shown  to  be  lacking  with  respect  to  usability. 

Issues  of  system  logic,  receiver  complexity,  and  display  design  were  among  the  critical 
human  factors  issues  expressed  by  pilots  as  lacking  on  most  GPS/LORAN  receiver  designs 
(Adams,  Adams,  Huntley,  &  Eldredge,  1992).  These  issues,  to  name  a  few,  combine  to 
determine  the  usability  of  a  system.  Typical  concerns  were:  non-intuitive  and  overly 
complex  menu  structures,  multifunction  knobs/keys  that  contribute  to  confusion  and 
forgetting,  and  a  lack  of  necessary  display  prompting. 

In  order  to  provide  design  recommendations  for  future  receivers,  current  receivers  must  be 
evaluated  and  design  inadequacies  must  be  identified.  The  purpose  of  this  study  was  to 
determine  the  conditions  that  contribute  to  pilot  GPS/LORAN  programming  errors  and  make 
design  recommendations  based  on  pilot  performance  and  subjective  reports. 

Five  pilots  rated  for  instrument  flight  (IFR)  participated  as  subjects.  Subjects  were  trained  to 
fly  a  simulator,  program  and  edit  a  flight  plan  on  a  common  GPS/LORAN  receiver,  and 
program  and  edit  the  receiver  while  flying  the  simulator.  Flight  turbulence  was  manipulated 
according  to  three  levels,  and  the  time  between  the  first  and  second  test  day  was  vari^  (7  or 
14  days).  Flight  performance  (crosstrack  error  and  altitude  deviation),  programming 
performance  (programming  time,  programming  errors,  and  recovery  time),  and  subjective 
reports  (pilot  questionnaires,  error  checklists,  and  general  comments)  were  collected  and 
analyzed. 

Flight  performance  data  indicated  that  subjects  were  well  trained  on  the  flight  simulation  task 
and  minimal  effects  of  turbulence  and  test  interval  were  found  across  test  session.  Pilot 
programming  performance  was  very  good  and  major  errors  were  only  found  on  13%  of  all 
flight/program  trials.  Although  few  errors  were  committed,  examination  of  the  conditions 
surrounding  each  error  indicated  that  display  design  and  system  logic  attributes  were 
contributory  factors.  In  addition,  examination  of  error  recovery  methods  showed  that  non- 
intuitive  menu  structures  complicated  and  confused  ihe  recovery  processes. 

Pilot  comments  were  sparsely  related  to  the  specific  errors  that  occurred  and  pilots  rarely 
recalled  "doing  anything  wrong"  after  an  error  was  made.  This  suggests  that  current  GPS/ 
LORAN  designs  are  opaque  and  do  not  provide  a  good  understanding  of  system  function  to 
the  pilot.  Some  pilots  developed  strategies  to  cope  with  the  demands  of  both  the  flight  and 
programming  tasks,  suggesting  that  pilots  were  attempting  to  gain  an  understanding  of  the 
GPS/LORAN  receiver  system  function. 


IX 


Recommendations  are  made  with  respect  to  receiver  controls,  functions,  menu  structure, 
display  design,  and  general  usability.  Future  research  issues  related  to  GPS  LORAN  use 
within  the  National  Airspace  System  and  the  effects  of  menu  structure,  error  recognition,  and 
error  recovery  are  discussed. 
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1.  INTRODUCTION 


Global  Positioning  System  (GPS)  technology  has  the  potential  to  enhance  pilot  performance 
by  providing  accurate  and  timely  navigation  information.  The  need  for  this  information  is 
great  for  general  aviation  pilots.  As  GPS  becomes  more  affordable  it  is  predicted  to  expand 
and  may  replace  present  navigation  systems  (e.g.,  Long-Range  Navigation  systems.  Instrument 
Landing  Systems).  Because  GPS  is  becoming  more  widely  used,  it  is  important  to  understand 
how  GPS  receivers  affect  pilot  performance  and  flight  safety. 

Preliminary  studies  indicate  that  current  GPS/LORAN  receivers  may  not  fully  support  pilot 
performance.  A  large  number  of  issues  were  identified  in  a  recent  survey  of  GPS/LORAN 
users  that  focus  on  system  usability  as  the  underlying  problem  area  (Adams  et  al.,  1992). 
System  logic,  complexity,  and  display  design  were  among  the  critical  human  factors  concerns 
that  affect  usability.  Examples  of  system  logic  difficulties  include  pilots  getting  lost  within 
the  menu  structure  associated  with  system  modes  and  functions.  The  organization  of 
information  pages  were  found  to  be  non-intuitive  and  cumbersome.  Pilots  reported  that 
searching  behaviors  and  pure  memorization  were  often  required  in  order  to  find  and  utilize 
receiver  functions.  These  behaviors  can  be  particularly  problematic  during  in-flight 
operations  because  they  contribute  to  "head-down"  time  and  hence,  have  the  potential  to 
compromise  flight  safety. 

The  high  degree  of  system  complexity  related  to  controls  and  hardware  issues  also  contribute 
to  degraded  usability.  Due  to  the  high  demand  for  panel  space,  system  designers  are  forced 
to  employ  the  use  of  multifunction  knobs  and  keys.  The  problem  with  this  approach  is  that 
often  times,  one  control  manipulation  (e.g.,  a  key  press)  will  have  many  different  and 
unrelated  functions  depending  upon  the  current  mode  of  the  receiver.  As  a  result,  it  is 
difficult  for  pilots  to  gain  a  good  understanding  about  the  relationships  between  control 
manipulation  and  system  function.  This  can  be  further  complicated  by  the  fact  that  most 
general  aviation  pilots  do  not  maintain  regular  flight  schedules  and  therefore  are  prone  to 
forgetting  over  short  periods  of  time  (Adams  et  al.,  1992). 

An  additional  concern  is  the  issue  of  display  design.  A  lack  of  good  system  logic  and 
structure  combined  with  high  complexity  creates  situations  in  which  pilots  are  unsure  about 
how  to  access  information  or  initiate  functions.  Adams  et  al.  (1992)  report  that  current 
receivers  are  lacking  with  respect  to  necessary  display  prompts.  That  is,  systems  do  not 
provide  information  about  the  required  steps  that  are  needed  to  accomplish  a  procedure  or 
function.  Furthermore,  these  receivers  often  do  not  provide  pilots  with  useful  error  messages. 
This  increases  the  likelihood  that  pilots  may  not  recognize  errors,  or  may  become  lost  during 
error  recovery  procedures  once  an  error  has  been  identified. 

Based  on  pilot  input,  it  is  clear  that  current  GPS/LORAN  receiver  designs  do  not  fully 
support  pilot  performance.  Given  that  these  systems  may  become  critical  for  precision 
navigation  tasks,  future  receiver  designs  must  consider  the  above  human  factors  concerns. 
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Perhaps  the  most  critical  issue  is  that  current  receiver  designs  have  the  potential  to  degrade 
performance,  particularly  in  single-pilot  aircraft. 


1.1  PRESENT  RESEARCH  ISSUES 

Field  studies  have  indicated  that  current  GPS/LORAN  receiver  designs  are  lacking.  If  pilots 
are  going  to  fully  benefit  from  new  and  existing  technology,  the  issues  of  usability  and 
receiver  design  must  be  investigated  in  the  laboratory.  The  difficulties  associated  with  current 
designs  must  be  systematically  examined  in  order  to  make  design  recommendations  for  the 
next  generation  GPS/LORAN  receiver. 

This  study  was  designed  to  achieve  two  goals.  One  goal  was  to  examine  the  issues  associated 
with  receiver  usability  and  to  make  design  recommendations  based  on  programming 
performance  and  subjective  reports.  A  second  goal  was  to  develop  a  methodology  for  future 
pilot-GPS  research. 

A  variation  of  the  secondary  task  method  (see  Wickens,  1 984)  was  used  to  examine 
GPS/LORAN  programming  performance.  Subjects  were  required  to  time-share  Hying  a 
simulator  with  programming  a  GPS/LORAN  receiver.  Flying  the  simulator  served  as  the 
primary  or  loading  task,  and  programming  the  receiver  served  as  the  secondary  task.  The 
turbulence  associated  with  the  flight  simulation  and  the  time  interval  between  test  sessions 
were  varied.  Thus,  programming  performance  should  vary  as  a  function  of  turbulence  and 
test  interval. 

Turbulence  (random  roll  and  pitch  inputs)  was  chosen  for  two  reasons.  First,  it  provides  a 
controlled  method  to  vary  the  attentional  demands  associated  with  the  flight  task.  Second,  the 
addition  of  turbulence  provides  a  level  of  difficulty  that  is  analogous  to  real-world  conditions. 
Although  the  turbulence  manipulation  is  not  completely  realistic,  it  does  require  subjects  to 
monitor  their  flight  instruments  as  they  would  be  expected  to  in  actual  turbulence  conditions. 
Attention  theories  (e.g.,  Kahneman,  1974;  Wickens,  1984)  postulate  that:  1)  Attention  is  a 
limited  commodity,  2)  tasks  compete  for  attention,  and  3)  degraded  performance  is  caused  by 
a  lack  of  sufficient  attention  to  perform  a  task.  Therefore,  it  was  predicted  that  GPS/LORAN 
programming  performance  should  degrade  as  turbulence  is  increased.  As  the  flight  task 
becomes  more  difficult,  less  attention  would  be  available  to  program  the  receiver  and 
performance  should  degrade. 

Test  interval  was  chosen  to  mimic  the  time  delay  that  typically  occurs  between  flights  among 
general  aviation  pilots.  All  subjects  received  one  training  session  and  two  test  sessions.  The 
effects  of  test  interval  were  examined  by  varying  the  amount  of  time  between  the  first  and 
second  data  collection  sessions.  One  group  of  subjects  returned  to  complete  the  second  test 
session  after  a  7-day  inteiYal;  a  second  group  returned  after  14  days.  It  was  predicted  that 
programming  performance  for  the  1 4-day  group  would  be  worse  than  the  7-day  group  because 
of  the  potential  for  increased  forgetting. 
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1.2  OBJECTIVES 


To  address  the  issues  outlined  above,  this  study  was  designed  to  investigate  pilot  GPS/ 
LORAN  programming  performance  as  a  function  of  turbulence  and  test  interval.  The 
objectives  of  the  study  were  to: 

1.  Examine  GPS/LORAN  programming  performance  during  simulated  flight  and  identify 
the  design  attributes  that  contribute  to  poor  performance. 

2.  Identify  which  aspects  of  receiver  programming  are  forgotten  over  time. 

3.  Examine  the  effects  of  increased  turbulence  on  programming  performance. 

4.  Provide  potential  solutions  to  the  interface  design  problems  discovered  in  Objectives  1 , 
2,  and  3. 

5.  Refine  the  experimental,  data  collection,  and  measurement  techniques  for  future  research 
on  GPS/LORAN  receiver  design. 
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2.  METHOD 


2.1  SUBJECTS 

Five  pilots  licensed  and  current  for  instrument  flight  (IFR)  participated  as  subjects.  All 
subjects  had  a  minimum  visual  acuity  of  20/20  (normal/corrected)  as  determined  by  a 
Graham-Field  eye  test  chart  (model  2867-1264).  Two  subjects  were  employees  of  the  Voipe 
National  Transportation  Systems  Center  (VNTSC)  and  were  given  an  account  number  to 
charge  their  time.  Three  subjects  unaffiliated  with  VNTSC  were  each  paid  $100.00  for 
participation. 


2.2  APPARATUS 

The  laboratory  was  equipped  with  two  banks  of  variable  level  fluorescent  lights.  Visual 
acuity  testing  was  performed  at  an  ambient  illumination  of  128  foot  candles  (approximately 
1377  lux).  The  experimental  trials  were  performed  at  an  ambient  illumination  of  5.6 
footcandles  (approximately  60  lux).  Luminance  was  measured  at  the  subject’s  approximate 
head  position  using  a  Sylvania  GTE  light  meter  (model  DS-2050).  The  major  equipment 
groups  were:  flight  simulator  and  controls,  GPS/LORAN  receiver,  and  audio/visual 
equipment;  each  are  described  below.  Figure  I  illustrates  the  layout  of  the  apparatus. 


2.2.1  FLIGHT  SIMULATOR 

The  flight  simulation  and  display  instruments  were  generated  using  a  486  MHz,  IBM 
compatible  micro-computer  with  Instrument  Flight  Trainer  software  (IFT-PRO  V5.1).  The 
flight  dynamics  and  instrument  display  were  configured  to  mimic  a  Cessna  172.  The  flight 
instrumentation  was  displayed  on  a  16"  NANAO  9080i  VGA  monitor  (model  #  MA  1660)  as 
shown  in  Figure  2.  The  flight  display  subtended  a  horizontal  visual  angle  of  30.4°  and  a 
vertical  visual  angle  of  24.2°  (assuming  a  viewing  distance  of  20.0  inches). 

The  primary  flight  instrumentation  included:  a  tum/slip  indicator,  airspeed  indicator,  attitude 
indicator,  directional  gyroscope,  altimeter,  vertical  speed  indicator,  and  a  very-high-frequency 
omni-directional  range  indicator  (with  course  deviation  needle).  The  outside  view  was  always 
gray  to  ensure  that  pilots  were  using  only  flight  instruments  to  receive  attitude  and  direction 
information.  A  description  of  all  flight  instruments  is  provided  in  Appendix  A. 

Flight  control  was  made  using  a  Novel  Twist  (model  #  NT-360)  flight  trainer  that  consisted 
of  a  control  panel,  yoke,  and  a  rudder  pedal  assembly.  Subjects  were  required  to  control 
flight  using  only  the  yoke  and  rudder  pedals.  The  control  yoke  provided  control  of  pitch  and 
roll.  The  range  of  pitch  control  was  6.4"  of  fore/aft  travel  with  a  control  resolution  of  16 
positions  per  inch.  Roll  control  ranged  from  +/-  90°  of  lateral  rotation  from  center  with 
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FIGURE  1 .  EXPERIMENTAL  APPARATUS 
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FIGURE  2.  FLIGHT  INSTRUMENTS  DISPLAY 


a  control  resolution  of  32  positions  per  90°  of  rotation.  Yaw  was  controlled  by  rudder 
pedals  that  had  a  range  from  +/-  3.25"  and  a  resolution  of  6  positions  per  inch. 

Under  certain  conditions,  subjects  were  required  to  use  the  "pause"  key  on  the  NT-360  panel 
to  terminate  a  trial.  Subjects  were  not  allowed  to  adjust  the  flight  attitude  by  using  the 
elevator  trim  or  throttle  controls.  An  exhaustive  description  of  the  NT-360  control  panel  can 
be  found  in  Brasington  (1990). 


2.2.2  GPS/LORAN  RECEIVER 

A  common  GPS/LORAN  receiver  was  mounted  approximately  33.5  inches  above  ground 
level  and  17.5  inches  to  the  right  of  the  NT-360  control  yoke.  This  placement  was  chosen  to 
mimic  typical  locations  found  in  general  aviation  aircraft.  An  arm  rest  (28.0  "  high  x  6.5  " 
wide)  was  provided  to  reduce  arm  fatigue. 

The  physical  dimensions  of  the  receiver  were  approximately  6.25  by  2.0  inches  in  width  and 
height,  respectively.  Figure  3  illustrates  the  receiver  face  plate.  TIk  unit  was  equipped  with 
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two  amber  colored,  light  emitting  diode  (LED)  displays.  Each  display  measured 
approximately  0.37S  inches  (height)  by  3.0  inches  (width).  The  secondary  di^lay  subtended 
a  horizontal  visual  angle  of  0.81**  and  a  vertical  visual  angle  of  6.46*’  (assuming  a  viewing 
distance  of  26.5  inches).  Each  display  was  controlled  with  two  concentric  control  knobs  and 
three  keys.  The  left-hand  set  of  knobs  and  keys  controlled  the  primary  display  and  the  right- 
hand  set  controlled  the  secondary  display. 
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FIGURE  3.  GPS/LORAN  RECEIVER  FACEPLATE 


In  general,  the  outer  knob  is  used  to  select  a  specific  function  and  the  inner  knob  is  used  to 
control  data  associated  with  the  selected  function.  Five  waypoint  categories  were  available 
on  each  side  of  the  receiver:  APT  (Airport),  VOR  (Very  High  Frequency  Omni-Directional 
Range  transmitter),  NDB  (Nondirectional  Radio  Beacon),  INT  (Intersections),  and  USER 
(User  defined  waypoints). 

The  cursor  (CRSR)  key  operates  as  an  on/off  toggle  to  assist  with  data  entry.  Pressing 
CRSR  once  begins  the  data  entry  process,  pressing  CRSR  again  terminates  data  entry. 

During  waypoint  entry,  the  inner  knob  is  used  to  scroll  characters  and  the  outer  knob  is  used 
to  control  cursor  position.  Users  may  also  scroll  entire  waypoint  identifiers  using  only  the 
inner  knob  once  a  category  is  selected.  The  acknowledge  (ACK)  key  has  many  fonctions. 
For  the  purpose  of  this  study,  it  was  primarily  used  to  begin  flight  plan  entry,  confirm 
waypoint  entry  and  editing  functions,  and  to  advance  positions  while  entering  a  flight  plan. 
The  ACK  key  prompts  users  by  flashing  on/off  during  the  flight  plan  entry  process. 


2.2.3  AUDIO/VISUAL  RECORDING  EQUIPMENT 

A  Panasonic  Digital  50(X)  System  video  camera  and  a  Panasonic  VHS  video  cassette  recorder 
(model  GX4  AG- 1950)  were  used  to  record  programming  behavior.  Audio  information  was 
recorded  using  a  Realistic  microphone  (model  33-1090  PZM),  an  Optimus  10-band  equalizer, 
and  a  Realistic  stereo-mixing  console. 
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2.3  EXPERIMENTAL  TASKS 


Subjects  performed  three  types  of  tasks:  the  flight  task,  the  GPS/LORAN  programming  task, 
and  the  combined  flight/program  task.  A  brief  description  of  each  of  these  tasks  follows. 

The  specific  order  and  number  of  trials  is  described  later  (see  the  PROCEDURE  section). 


2.3.1  FLIGHT  TASK 

The  goal  of  the  flight  task  was  to  maintain  aircraft  altitude  and  heading  on  a  preset,  straight 
and  level  flight  path.  The  course  heading  was  set  to  292  degrees  and  subjects  were  not 
explicitly  given  this  information.  Subjects  were  instructed  to  keep  the  course  deviation  needle 
centered  on  the  number  one  VOR  while  maintaining  altitude  to  within  +/-  100  feet  of  the 
preset  5000  foot  value.  Airspeed  was  preset  at  1(X)  knots  and  varied  only  with  aircraft 
attitude  changes;  subjects  were  not  allowed  to  manipulate  the  throttle  control. 

The  turbulence  a.ssociated  with  each  flight  trial  was  manipulated  by  varying  the  turbulence 
parameter  of  the  flight  simulation  software.  Three  turbulence  levels  (low.  medium,  and  high) 
were  chosen  and  verified  by  subject  matter  experts  as  providing  distinct  levels  of  flight 
difficulty.  The  turbulence  level  assigned  to  a  .specific  trial  did  not  change  within  a  trial. 

Each  single  task  flight  trial  lasted  four  minutes  and  was  terminated  by  the  experimenter. 
Crosstrack  error  and  altitude  deviation  measures  were  displayed  on  the  subjects  monitor  at  the 
end  of  each  trial  to  provide  feedback  and  enhance  subject  motivation. 


2.3.2  GPS/LORAN  RECEIVER  PROGRAMMING  TASK 

The  GPS/LORAN  receiver  was  programmed  to  run  in  simulation  mode  and  did  not  provide 
dynamic  position  information.  The  programming  task  consisted  of  three  parts:  1 )  Program  a 
four-waypoint  flight  plan,  2)  edit  the  flight  plan,  and  3)  check  the  accuracy  of  the  flight  plan. 
Figure  4  depicts  the  stages  of  the  programming  task  and  Appendix  B  shows  a  sample  flight 
plan. 

Each  flight  plan  consisted  of  four-waypoint  identifier  names  and  their  respective  waypoint 
categories  (e.g.,  VOR,  APT,  USER,  INT,  or  NDB).  The  flight  plans  were  developed  using 
sectional  charts  and  were  designed  to  include  different  categories  throughout  each  position  in 
the  flight  plan.  An  unique  flight  plan  was  developed  for  each  receiver  programming  trial. 
Each  flight  plan  was  entered  into  the  GPS/LORAN  receiver  prior  to  the  experiment  to  ensure 
that  all  waypoints  were  contained  in  the  receiver  databa.se. 

Figure  4  illustrates  the  basic  steps  required  to  enter  a  flight  plan.  Waypoint  entry  could  be 
achieved  by  using  either  the  scrolling  or  cursor  entry  methods  (see  Appendix  C  for  waypoint 
entry  procedures).  Note  that  steps  2-4  are  repeated  until  the  specific  number  of  waypoints  are 
entered  into  the  flight  plan.  At  step  5,  subjects  are  required  to  exit  and  enter  the  flight 


plan  mcxie  by  turning  the  large,  primary  control  knob  one  detent  and  back.  At  this  point,  the 
secondary  display  shows  the  flight  plan  leg  (e.g..  1st  leg.  2nd  leg)  and  the  primary  display 
.shows  the  waypoints  of  that  leg  (e.g..  SXD  n  to  EEN  v). 

Next,  the  editing  sequences  (at  step  6.  10.  or  14)  were  performed  and  always  involved 
inserting,  deleting,  and  changing  waypoints.  The  order  of  these  functions  was 
counterbalanced  across  trial.  When  the  editing  portion  of  the  task  was  completed,  subjects 
were  a.sked  to  check  the  accuracy  (step  18)  of  the  flight  plan  by  consulting  the  flight  plan 
printed  on  the  reverse  side.  Although  the  single  task  programming  trials  were  self-paced, 
subjects  were  encouraged  to  program  as  quickly  as  possible  but  were  required  to  maintain 
100%  accuracy.  Short-cuts  to  obtain  a  correct,  final  flight  plan  were  discouraged.  That  is. 
subjects  were  explicitly  told  to  follow  the  exact  sequence  as  printed  on  the  flight  plan. 


2.3.3  FLIGHT/PROGRAM  TASK 

The  flying  and  programming  tasks  were  combined  in  these  trials.  Subjects  were  asked  to 
maintain  the  flight  in  the  same  manner  as  during  the  single  task  flight  trials  in  addition  to 
programming,  editing,  and  checking  a  flight  plan.  Verbal  instructions  stressed  that  primary 
emphasis  must  be  given  to  the  flight  portion  of  the  task.  The  initial  flight  parameters  were 
exactly  the  same  as  tho.se  used  during  flight  training.  Subjects  were  instructed  to  maintain 
their  altitude  at  ±100  ft,  and  u.se  the  GDI  to  maintain  their  course  as  accurately  as  possible. 

Prior  to  the  start  of  each  trial,  subjects  were  given  a  flight  plan,  face-down  on  the  kneeboard. 
Subjects  were  allowed  to  commence  programming  after  thirty  seconds  of  the  trial  had  elapsed 
(signaled  by  the  experimenter).  This  30  .second  period  allowed  subjects  the  opportunity  to 
establish  a  relatively  stable  flight  profile. 

Subjects  were  responsible  for  terminating  the  flight/program  trials,  after  verifying  the 
accuracy  of  the  flight  plan,  by  pressing  the  pre-designated  "pause"  button  on  the  NT-360 
control  panel.  The  trial  duration  and  flight  performance  measures  were  then  recorded  by  the 
PC  and  performance  feedback  was  di.splayed  on  the  subject’s  monitor  in  place  of  the  flight 
instrument  display. 


2.4  EXPERIMENTAL  DESIGN 

A  mixed  experimental  design  was  chosen  to  examine  the  effects  of  turbulence  and  test 
interval  on  GPS/LORAN  programming  performance  (see  Table  I ).  The  within-subjects 
independent  variables  were  task  (flight  only  or  flight/program)  and  turbulence  (low,  medium, 
or  high).  The  time  interval  between  the  first  and  second  test  days  (7  or  1 4  days)  was  initially 
chosen  as  the  between-subjects  independent  variable.  Due  to  time  constraints,  three  subjects 
completed  the  14-day  test  interval  condition  and  two  subjects  completed  the  7-day  condition. 
As  a  result,  the  between-subjects  variable  of  test  interval  was  collapsed  to  repre.sent  a  "global" 
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factor  of  test  day.  That  is,  the  data  recorded  on  the  7th  or  14th  day  were  uniformly  treated  as 
the  second  test  day  data. 


TABLE  1.  EXPERIMENTAL  DESIGN 


Training  Session 

Test  Day  1 

Test  Day  2  | 

Fly  or  Program 

Fly  or  Program 
(warm-up) 

Fly  Only 
(warm-up) 

Program  or  Fly 

Program  or  Fly 
(warm-up) 

Fly/Program 

BREAK 

BREAK 

Fly/Program 

Fly/Program 

Fly/Program 

BREAK 

Fly  or  Program 

Fly/Program 

Fly/Program 

BREAK 

BREAK 

Fly/Program 

Program  or  Fly 

Fly/Program 

Fly/Program 

Fly/Program 

2.5  PROCEDURE 

All  subjects  performed  one  training  session  and  two  test  sessions.  The  first  test  session  for 
all  subjects  was  completed  one  day  following  training.  Subjects  were  randomly  assigned  to 
either  test  interval  condition  (7  or  14  days)  for  the  second  test  session.  Detailed  procedures 
and  instructions  are  shown  in  Appendix  D;  each  session  is  described  below. 


2.5.1  TRAINING  SESSION 

After  completing  the  visual  acuity  test,  each  subject  read  and  signed  an  informed  consent 
form  that  summarized  the  purpose  and  general  procedures  of  the  experiment  (see  Appendix 
E).  Next,  subjects  completed  the  pilot  experience  questionnaire  (see  Appendix  F).  This 
questionnaire  was  designed  to  obtain  demographic  data,  flight  experience,  and  previous 
GPS/LORAN  experience.  When  applicable,  subjects  were  asked  to  rate  the  performance 
characteristics  associated  with  previously  operated  GPS/LORAN  receivers  and  provide  general 
design  recommendations. 
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Subjects  were  randomly  assigned  to  a  counterbalanced  task  condition  sequence.  The  t)rdcr  of 
single  task  conditions  and  the  turbulence  levels  for  the  dual  task  conditions  were 
counterbalanced.  Subjects  were  familiarized  with  either  the  programming  task  or  the  flight 
task,  depending  upon  the  task  order  sequence  they  were  assigned.  Two  subjects  performed  a 
block  (three  trials)  of  programming  trials  first,  followed  by  a  block  of  flight  trials.  Three 
subjects  performed  a  block  of  flight  trials  first,  followed  by  a  block  of  programming  trials. 
For  the  single  task  flight  and  dual  task  flight/program  trials  each  turbulence  level  was 
performed  once  in  a  block  and  always  progressed  from  low  to  high  to  facilitate  learning. 
After  single  task  practice,  subjects  performed  a  block  of  flight/program  trials. 

This  .sequence  was  repeated  in  the  .same  order  for  the  second  half  of  the  Training  Session. 
Breaks  lasting  approximately  10  minutes  were  provided  after  every  second  bliKk  of  trials. 
Overall,  subjects  performed  18  trials  during  the  Training  Session  (6  program  trials.  6  flight 
only  trials,  and  6  flight/program  trials;  2  repetitions  at  each  turbulence  level). 


2.5.2  TEST  DAY  1 

Subjects  performed  a  block  of  programming  trials  followed  by  a  block  of  flight  trials,  or  the 
reverse,  to  serve  as  warm-up.  The  task  order  was  identical  to  that  of  the  training  session  for 
a  given  subject.  Questions  concerning  the  GPS/LORAN  receiver  could  be  answered  only 
during  warm-up  trials,  after  this  period  subjects  were  forced  to  remedy  problems  on  their 
own. 

Following  the  warm-up  trials,  subjects  performed  four  blocks  of  flight/program  trials 
consisting  of  three  trials  per  block.  Each  turbulence  level  was  performed  once  per  block  in  a 
counter-balanced  order.  If  a  programming  error  occurred,  subjects  were  questioned  about  the 
nature  of  the  error  immediately  following  the  trial.  A  checklist  of  possible  responses  was 
used  for  prompting  (see  Appendix  D)  and  subjects  were  given  the  opportunity  to  elaborate  on 
the  reasons  behind  the  errors.  Two  10  minute  breaks  were  provided  throughout  the  session. 
Overall,  subjects  performed  18  trials  for  Test  Day  1  (3  programming  trials,  3  flight  trials,  and 
12  flight/programming  trials;  4  repetitions  at  each  turbulence  level). 

2.5.3  TEST  DAY  2 

Subjects  returned  in  either  7  or  1 4  days,  depending  on  the  group  they  were  assigned  to.  and 
performed  two  single-task  flight  trials  for  warm-up.  To  keep  the  overall  number  of  trials  low 
without  compromising  warm-up,  both  the  low  and  high  turbulence  conditions  were  performed. 
The  first  trial  was  always  a  low  turbulence  trial  and  the  second  was  always  a  high  turbulence 
trial.  No  receiver  programming  warm-up  was  given  and  subjects  were  not  provided  any 
assistance  from  the  experimenter  concerning  the  programming  task.  Following  the  flight  task 
warm-up  trials,  subjects  performed  four  flight/program  blocks  in  the  same  order  as  during 
Test  Day  I . 
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Subjects  were  questioned  about  any  programming  errors  using  the  same  checklist  as  in  Test 
Day  1,  and  a  10  minute  break  was  provided  midway  through  the  session.  Overall,  subjects 
performed  14  trials  for  Test  Day  2  (2  flight  trials,  and  12  flight/programming  trials;  4 
repetitions  at  each  turbulence  level).  When  all  trials  were  completed,  subjects  were  asked  to 
complete  a  post-experiment  questionnaire  (see  Appendix  G).  The  purpose  of  this 
questionnaire  was  to  allow  subjects  to  elaborate  on  any  positive  or  negative  aspects  of  the 
GPS/LORAN  receiver.  Subjects  were  also  asked  to  list  any  desired  navigation  information 
that  a  GPS/LORAN  receiver  should  display  and  to  provide  design  recommendations 
concerning  the  knobs/keys.  labeling,  displays,  and  procedures. 


2.6  PERFORMANCE  MEASURES 


2.6.1  FLIGHT  PERFORMANCE 

The  difference  between  the  aircraft's  position  and  the  centerline  of  the  desired  course  (i.e., 
crosstrack  error)  was  measured  in  feet.  Crosstrack  error  was  sampled  once  per  second  and 
the  root  mean  square  (RXTK)  for  each  trial  was  computed.  The  difference  between  the 
aircraft’s  altitude  and  the  desired  altitude  (i.e.,  altitude  deviation)  was  measured  in  feet. 
Altitude  was  sampled  once  per  second  and  the  root  mean  square  (RALD)  was  computed  for 
each  trial. 


2.6.2  PROGRAMMING  PERFORMANCE 

The  video  recordings  were  examined  to  provide  information  about  both  error  and  error-free 
programming  performance.  To  determine  the  programming  time  associated  with  each  part  of 
the  programming  task,  the  total  time-on-task  was  divided  into  three  segments.  The  time  to 
enter  a  flight  plan  was  defined  as  the  beginning  of  each  flight/program  trial  (i.e.,  the  30 
second  mark  signaled  by  the  experimenter)  up  to  the  final  correct  waypoint  entry.  This 
signalled  the  beginning  time  for  the  editing  task  which  ended  when  the  final  waypoint  was 
successfully  edited.  The  remaining  time  up  to  the  end  of  the  trial  was  categorized  as 
accuracy  checking  time.  These  times  were  .summed  to  provide  a  measure  of  total 
programming  time  for  the  error-free  trials. 

A  second  goal  was  to  determine  the  conditions  under  which  errors  occurred  and  to  examine 
error  recovery.  An  error  was  defined  as  a  control  manipulation  that  interrupted  the  progress 
of  the  programming  task  and  required  additional  control  manipulations  in  order  to  return  to  an 
error-free  state.  Error  time  was  defined  as  the  point  at  which  an  incorrect  control 
manipulation  was  initiated  until  recovery  from  the  error  was  achieved.  The  steps  required  to 
recover  were  analyzed  to  provide  information  about  the  subjects’  knowledge  of  system  logic 
and  to  evaluate  the  design  attributes  of  the  receiver. 
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2.6.3  QUESTIONNAIRES 


The  pilot  experience  questionnaire  was  designed  to  gather  demographic  and  qualitative  data, 
and  to  solicit  design  recommendations  based  on  previous  GPS/LORAN  experience  (sec 
Appendix  F).  All  closed-end  questions  were  rated  on  a  7-point  Likert-type  scale,  where: 

1  =  "very  easy,"  2  =  "easy,"  3  =  "somewhat  easy."  4  =  "neutral."  5  =  "somewhat  ditTiculi." 
6  =  "difficult,"  and  7  =  "very  difficult." 


The  post-experiment  questionnaire  was  designed  to  gather  the  negative  and  positive  aspects  (if 
any)  as.sociated  with  the  programming  task  (see  Appendix  G).  Subjects  were  also  asked  to 
provide  .specific  design  recommendations  based  on  experience  gained  in  the  present  study.  In 
addition,  comments  related  to  programming  errors  and  general  comments  about  the 
experiment  were  collected  and  analyzed. 
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3.  RESULTS 


The  results  described  below  are  grouped  according  to  six  sections.  Pilot  flight  and 
GPS/LORAN  experience  are  reported  based  on  the  responses  provided  on  the  pilot  experience 
questionnaire.  Flight  performance  measures  recorded  for  the  flight  only  and  flight/ program 
conditions  are  described  in  the  next  section.  The  programming  performance  measured  in  the 
flight/program  trials  of  the  first  and  second  test  days  are  divided  into  an  error-free 
performance  section  and  a  programming  errors  section.  It  is  important  to  note  that  because 
only  five  subjects  completed  the  study,  limited  significance  testing  was  performed  on  the 
performance  data.  Mean  and  median  scores  are  reported  to  examine  trends  in  the  data. 

Lastly,  subjective  data  are  reported  in  the  post-questionnaire  and  general  comments  sections. 


3.1  PILOT  FLIGHT  AND  GPS/LORAN  EXPERIENCE 

Subjects’  ages  ranged  from  29  to  62  years,  with  a  median  age  of  50  years.  Total  flight  time 
(actual  and  simulated  flight)  ranged  from  300  to  21,000  hours,  with  a  median  of  1,100  hours. 
Total  IFR  time  (actual  and  simulated  flight)  ranged  from  60  to  700  hours,  with  a  median  of 
218  hours.  All  subjects  reported  having  general  aviation  experience,  one  subject  reported 
having  both  Part  121  and  Part  135  experience.  The  types  of  aircraft  that  subjects  were 
currently  experienced  with  were;  Cessna  172/210,  Beechcraft  58/76/Duchess,  Piper  28,  and 
an  Aerostar  700. 

Three  of  five  subjects  had  prior  experience  with  either  the  ARNAV  R50i,  ARNAV  FMS,  and 
the  KLN  88  type  LOR  AN  receivers.  The  ratings  associated  with  the  usability  and  the  design 
recommendations  for  each  receiver  are  provided  below. 

The  estimated  time  in  use  during  flight  for  the  ARNAV  FMS  was  10%.  The  readability  of 
the  display  panel,  flight  plan  creation,  flight  plan  editing,  controls  operation,  and  overall 
usability  were  each  rated  as  "easy."  Error  recovery  was  rated  as  "very  easy."  Waypoint  entry 
was  reported  as  causing  a  brief  distraction  from  primary  flight  duties  and  no  design  changes 
were  recommended  for  this  receiver. 

The  readability  and  error  recovery  of  the  ARNAV  R50i  were  each  rated  as  "very  easy." 

Flight  plan  creation,  editing,  and  overall  usability  were  each  rated  as  "easy",  and  controls 
operation  was  rated  as  "somewhat  easy."  This  receiver  was  estimated  as  being  used  100%  of 
the  time.  Operation  of  this  receiver  was  rated  as  non-distracting  from  primary  flight  duties, 
however,  the  "Direct-To-Waypoint"  procedure  was  rated  as  hard  to  remember. 

The  KLN  88  was  estimated  as  being  used  90%  of  the  time.  The  overall  usability  and  display 
readability  were  each  rated  as  "very  easy."  Flight  plan  creation  and  editing  were  each  rated 
as  "somewhat  difficult",  and  error  recovery  and  controls  operation  were  each  rated  as 
"neutral."  This  receiver  was  rated  as  non-distracting  from  primary  flight  duties  and  the  only 
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design  recommendation  was  for  better  documentation.  It  was  suggested  that  flow  chart 
presentation  may  simplify  the  procedures  documentation. 


3.2  FUGHT  PERFORMANCE 

The  root  mean  square  crosstrack  error  (RXTK)  and  root  mean  square  altitude  deviation 
(RADV)  scores  obtained  for  each  trial  were  averaged  across  condition  (Task  x  Turbulence  x 
Session).  The  RXTK  data  are  listed  in  Table  2  and  illustrated  in  Figure  5.  Crosstrack 
performance  was  best  for  all  single  flight  task  conditions.  Performance  improved  for  the  low 
and  high  turbulence  conditions  across  session  and  was  essentially  equal  during  the  warm-up 
trials  of  the  second  test  day.  Crosstrack  error  while  programming  was  more  variable  during 
the  training  session  but  stabilized  for  the  low  and  medium  turbulence  conditions  during  the 
first  and  second  test  days.  Crosstrack  error  increased  (non-significantly)  on  Test  Day  2  for 
the  high  turbulence  condition. 


TABLE  2.  ROOT  MEAN  SQUARE  CROSSTRACK  ERROR 
BY  TASK.  TURBULENCE,  AND  SESSION 


Task 

Turbulence 

Session 

Training 

Test  Day  1 

Test  Day  2 

Fly  Only 

Low 

137.87 

(62.46) 

73.90 

(19.39) 

63.94 

(7.93) 

Fly  Only 

Medium 

120.27 

(58.61) 

132.50 

(71.48) 

N/A 

Fly  Only 

High 

103.41 

(50.06) 

106.56 

(44.99) 

60.88 

(0.67) 

Fly  and 
Program 

Low 

347.07 

(222.31) 

260.95 

(219.76) 

240.78 

(227.82) 

Fly  and 
Program 

Medium 

283.09 

(216.35) 

280.55 

(199.20) 

234.43 

(136.10) 

Fly  and 
Program 

High 

443.19 

(276.87) 

253.94 

(176.32) 

293.17 

(195.46) 

Note:  Standard  deviations  are  given  in  parentheses.  No  warm-up  trials  were 
performed  for  the  medium  turbulence  condition  of  the  second  test  day. 
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FIGURE  5.  ROOT  MEAN  SQUARE  CROSSTRACK  ERROR  AS  A 
FUNCTION  OF  TASK.  TURBULENCE.  AND  SESSION 


The  RADV  data  are  listed  in  Table  3  and  illustrated  in  Figure  6.  The  RADV  scores  show  a 
trend  similar  to  the  RXTK  scores  in  that  performance  under  single  task  conditions  was 
superior  to  all  dual  task  conditions.  Altitude  deviation  performance  improved  on  the  first  test 
day  for  the  low  and  medium  turbulence  conditions  and  remained  stable  for  Test  Day  2.  Dual 
task  performance  improved  across  session  for  all  levels  of  turbulence.  Perhaps  the  most 
dramatic  improvement  was  the  decrease  in  altitude  deviation  across  test  days  for  the  high 
turbulence  condition.  Post-hoc  F-tests  for  simple  effects  revealed  that  performance  was 
significantly  better  for  the  low  (E  (1,  12)  =  20.21,  p  <  0.01)  and  medium  CE  (1,12)  = 
15.93,  p  <  0.01)  turbulence  conditions  of  Test  Day  1. 


TABLE  3.  ROOT  MEAN  SQUARE  ALTITUDE  DEVIATION 
BY  TASK,  TURBULENCE,  AND  SESSION 


Session 

Task 

Turbulence 

Training 

Test  Day  1 

Test  Day  2 

Fly  Only 

Low 

22.79 

12.80 

14.84 

(7.02) 

(4.01) 

(4.80) 

Fly  Only 

Medium 

22.28 

14.14 

N/A 

(5.91) 

(3.38) 

Fly  Only 

High 

18.87 

17.12 

19.80 

(5.80) 

(6.51) 

(7.80) 

Fly  and 

Low 

37.28 

32.20 

32.99 

Program 

(13.78) 

(10.53) 

(15.33) 

Fly  and 

Medium 

50.12 

33.75 

28.39 

Program 

(34.92) 

(11.33) 

(12.52) 

Fly  and 

High 

47.67 

46.04 

33.40 

Program 

(9.57) 

(48.32) 

(16.29) 

Note:  Standard  deviations  are  given  in  parentheses.  No  warm-up  trials  were 
performed  for  the  medium  turbulence  condition  of  the  second  test  day. 
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FIGURE  6.  ROOT  MEAN  SOUARE  ALTITUDE  DEVIATION  AS  A 
FUNCTION  OF  TASK,  TURBULENCE,  AND  SESSION 


3.3  ERROR-FREE  GPS/LORAN  PROGRAMMING  PERFORMANCE 

Programming  performance  from  the  flight/program  trials  of  Test  Days  1  and  2  were 
examined  to  determine  the  effects  of  turbulence  and  test  interval.  The  training  and  warm-up 
trials  were  omitted  from  all  programming  performance  analyses.  Subjects  performed  104  of 
120  possible  trials  without  an  error  for  an  overall  accuracy  rating  of  87%. 

The  mean  programming  times  according  to  test  day  (one  or  two),  programming  task  phase 
(entry,  edit,  accuracy  check,  and  total  time),  and  turbulence  level  (low,  medium,  and  high) 
are  provided  in  Table  4,  Figure  7  shows  the  mean  time  required  to  complete  all  phases  of 
the  programming  task  as  a  function  of  test  day  and  turbulence.  On  the  average, 
programming  time  increased  linearly  across  the  low  (M  =  304.23),  medium  (M  =  361.65), 
and  high  (M  =  374.33)  turbulence  levels.  The  effect  of  test  day  was  nearly  uniform  for 
each  level  of  turbulence  with  the  exception  of  the  high  turbulence  condition  of  the  second  test 
day.  Subjects  required  slightly  more  time  (non-significant)  at  the  highest  turbulence  level  of 
the  second  test  day  (M  =  384.5)  as  compared  with  the  first  test  day  (M  =  364.16). 
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TABLE  4.  PROGRAMMING  TIME  FOR  ERROR-FREE  TRIALS 
BY  TEST  DAY.  TASK  PHASE,  AND  TURBULENCE 


Test  Day 

Task  Phase 

Turbulence  | 

Low 

Medium 

High 

1 

Entry 

123.44 

(35.21) 

171.33 

(62.02) 

164.53 

(65.58) 

1 

Edit 

153.50 

(63.52) 

163.93 

(40.45) 

167.00 

(70.05) 

1 

Accuracy 

Check 

29.72 

(14.31) 

27.27 

(12.32) 

32.63 

(18.22) 

1 

Total 

Time 

306.67 

(99.25) 

362.53 

(98.60) 

364.16 

(112.03) 

2 

Entry 

141.33 

(60.22) 

172.72 

(70.09) 

181.79 

(65.32) 

2 

Edit 

140.00 

(56.21) 

161.89 

(97.65) 

176.71 

(75.79) 

2 

Accuracy 

Check 

20.44 

(12.28) 

26.17 

(14.82) 

26.00 

(18.25) 

2 

Total 

Time 

301.78 

(112.34) 

360.78 

(165.61) 

384.50 

(127.95) 

Note.  All  times  are  given  in  seconds.  Standard  deviations 
are  given  in  parentheses. 


In  general,  no  appreciable  performance  differences  by  task  phase  were  found.  Examination 
of  Table  4  shows  that  both  flight  plan  entry  and  editing  times  increased  as  turbulence 
increased.  Higher  turbulence  levels  required  more  time  to  complete  the  task.  In  addition, 
Table  4  shows  tiiat  subjects  required  less  time  to  complete  the  accuracy  checking  phase  on 
the  second  test  day. 
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FIGURE  7.  ERROR-FREE  PROGRAMMING  PERFORMANCE  AS  A  FUNCTION  OF 
TURBULENCE  AND  TEST  INTERVAL 


3.4  GPS/LORAN  PROGRAMMING  ERRORS 

Errors  occurred  on  16  of  120  possible  trials  (13%).  Table  5  lists  the  distribution  of  errors 
according  to  test  day  (one  or  two),  task  phase  (flight  plan  entry  or  editing),  and  turbulence 
level  (low,  medium,  or  high).  Generally,  subjects  committed  more  errors  as  turbulence 
increased.  The  fewest  number  of  errors  occurred  at  the  lowest  turbulence  level  (n=3)  as 
compared  to  the  medium  (n=6)  and  high  (n=7)  turbulence  levels.  Seven  errors  were 
committed  on  the  first  test  day  and  nine  were  committed  on  the  second  test  day.  Turbulence 
had  a  greater  impact  on  performance  of  the  second  test  day  in  which  6  out  of  9  errors 
occurred  at  the  highest  turbulence  level.  Overall,  more  than  twice  as  many  errors  occurred 
during  flight  plan  entry  (n=ll)  as  compared  to  flight  plan  editing  (n=5). 

The  errors  were  grouped  according  to  the  conditions  surrounding  each  error.  Four  distinct 
categories  were  developed  and  each  contained  repeat  occurrences.  When  possible,  the 
recovery  method  and  recovery  time  associated  with  each  error  was  examined.  Levels  of 
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TABLE  5.  PROGRAMMING  ERROR  FREOUENCY  BY  TEST  DAY,  TASK  PHASE. 
AND  TURBULENCE 


Test  Day 

Task  Phase 

Turbulence 

Total 

Low 

Medium 

High 

1 

Entry 

1 

2 

1 

4 

_ _ 

Edit 

1 

2 

0 

3  1 

1  Total 

2 

4 

1 

7  1 

2 

Entry 

1 

2 

4 

7  1 

2 

Edit 

0 

0 

2 

2  1 

Total 

1 

2 

6 

9  1 

Total 

3 

6 

7 

16  1 

confusion  were  recorded  subjectively  (e.g.,  low,  medium,  and  high)  to  provide  additional 
information  about  the  error  commission  and  recovery  processes. 

The  Misspell  Waypoint  error  did  not  involve  a  recovery  process.  In  this  case,  the 
appropriate  waypoint  category  was  selected  but  the  waypoint  identifier  was  misspielled.  The 
misspelled  waypoint  was  contained  in  the  database  and  ^erefore  was  accepted  by  the  GPS/ 
LORAN  receiver.  This  error  occurred  twice  and  no  recovery  attempts  were  made  during  the 
flight  plan  entry  phase  of  the  task.  This  error  was  not  noticed  during  the  accuracy  checking 
phase  of  the  task  because  the  waypoint  was  deleted  during  the  editing  phase  as  instructed. 

The  correct  waypoint  identifier  was  "JIMIE"  and  the  entered  waypoint  was  "JIMMI." 

Recoveries  were  made  for  the  remaining  three  error  categories.  Only  the  Select  Incorrect 
Waypoint  Category  error  did  not  involve  use  of  the  acknowledge  (ACK)  key.  The  conditions 
surrounding  each  error,  impact  of  the  error,  recovery  method,  mean  recovery  time,  and 
frequency  of  occurrence  are  listed  in  Table  6.  Each  category  is  described  in  a  separate 
section  below  and  the  mean  recovery  times  for  each  category  are  illustrated  in  Figure  8. 
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TABLE  6.  GPS/LORAN  PROGRAMMING  ERRORS  CLASSIFICATION  SCHEME 
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FIGURE  8.  MEAN  RECOVERY  TIME  BY  ERROR  CATEGORY 


3,4.1  SELECT  INCORRECT  WAYPOINT  CATEGORY  ERROR 

The  Select  Incorrect  Waypoint  Category  error  occurred  when  subjects  selected  an  incorrect 
waypoint  category  (e.g.,  APT,  VOR,  NDB,  INT,  or  USER),  but  did  not  attempt  to  enter  a 
waypoint.  Figure  9  illustrates  the  steps  taken  when  this  error  occurred.  This  error  occurred 
once  at  each  turbulence  level.  The  common  method  of  recovery  was  to  exit  and  re-enter  the 
flight  plan  mode  using  the  large  primary  knob.  This  action  (see  step  3  of  Figure  9) 
essentially  cleared  any  unacknowledged  information  and  allowed  programming/editing  to 
continue  without  a  complete  restart.  Because  the  error  was  recognized  before  waypoint  entry 
was  attempted,  recovery  was  quick  (M  =  20.33)  and  relatively  easy. 
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FIGURE  9.  SELECT  INCORRECT  WAYPOINT  CATEGORY  ERROR 


3.4.2  DEFAULT  WAYPOINT  ENTRY  ERROR 

When  subjects  pressed  the  acknowledge  (ACK)  key  after  selecting  a  category  but  prior  to 
entering  an  identifier,  a  default  waypoint  was  entered  into  the  flight  plan.  This  occurrence 
was  classified  as  a  Default  Waypoint  Entry  error.  Figure  10  illustrates  the  stqis  taken  when 
this  error  occurred.  This  error  occurred  three  times  at  the  medium  and  twice  at  the  high 
turbulence  levels.  All  Default  Waypoint  Entry  errors  occurred  during  the  flight  plan  entry 
phase. 

One  recovery  method  for  the  Default  Waypoint  Entry  error  was  to  edit  the  flight  plan  using 
the  delete  function  and  resume  programming  (see  stqis  1>S  and  10  of  Figure  10).  One 
subject  recognized  the  error  immediately,  did  not  show  any  confusion,  and  corrected  the 
error  in  8  seconds  using  this  recovery  method. 

For  the  remaining  four  occurrences,  confusion  ranged  from  medium  to  high  and  the  mean 
recovery  time  was  128.25  seconds.  In  these  cases,  the  recovery  method  included  the 
execution  of  random  knob  turns  and  key  presses  until  the  editing  functions  were  found  (see 
steps  1-4  and  6-10  of  Figure  10).  On  two  occurrences,  ACK  was  pressed  multiple  times  and 
multiple  default  waypoints  were  added  prior  to  recovery.  The  mean  recovery  time  for  all 
Default  Waypoint  ^try  trials  was  104.2  seconds. 
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FIGURE  10.  DEFAULT  WAYPOINT  ENTRY  ERROR 


3.4.3  ACKNOWLEDGE  INVALID  WAYPOINT  ERROR 


When  subjects  entered  an  incorrect  category  and/or  identifier  and  pressed  the  ACK  key,  't 
was  classified  as  an  Acknowledge  Invalid  Waypoint  error.  This  error  is  a  combination  of  the 
Select  Incorrect  Waypoint  Category  and  Default  Waypoint  Entry  errors.  These  actions 
produced  a  "Saved  Position"  mes.sage  on  the  .secondary  display  and  the  ACK  key  became 
steadily  illuminated.  Figure  1 1  illustrates  the  steps  taken  when  this  error  occurred. 

In  one  separate  instance,  the  ACK  key  was  pres.sed  prior  to  selecting  a  waypoint  category 
when  the  "Undefined  Leg"  screen  was  displayed.  This  occurred  on  the  first  trial  of  the 
second  Test  Day  and  was  clearly  a  result  of  forgetting.  The  subject  reported  having  difficulty 
remembering  how  to  begin  the  programming  process.  The  "Saved  Position"  message  was 
displayed  and  the  subject  recovered  by  randomly  turning  knobs  and  pressing  keys  until  the 
message  was  erased  as  described  below. 

For  all  occurrences  of  this  error,  high  levels  of  confusion  were  observed  and  subjects  were 
unsure  of  how  to  recover.  This  was  evident  by  noting  that  every  case  involved  random 
control  manipulations  prior  to  recovery.  The  only  way  to  recover  was  to  press  ACK  one  time 
after  the  "Saved  Position"  message  was  displayed.  This  occurred  twice  with  a  mean  recovery 
time  of  126.5  seconds  (see  steps  1-4  and  9  of  Figure  1 1).  For  the  remaining  trials,  ACK  was 
pressed  multiple  times  and  multiple  "Saved  Position"  mes.sages  occurred  prior  to  recovery 
(see  steps  1-9  of  Figure  1 1).  The  mean  recovery  time  for  the  multiple  occurrences  was  175.0 
seconds. 


3.5  POST-EXPERIMENT  QUESTIONNAIRE 

The  specific  responses  for  each  question  are  shown  in  Appendix  H.  a  summary  is  provided 
below. 

Question  1  asked  subjects  to  describe  the  programming  procedure(s)  that  were  found  to  be 
most  confusing.  One  subject  responded  that  it  seemed  easier  to  misidentify  waypoints  during 
editing.  One  subject  responded  that  better  layout  of  the  LED  display  was  needed,  and  three 
subjects  replied  that  nothing  was  confusing. 

Question  2  asked  subjects  to  describe  the  single  best  feature  of  the  GPS/LORAN.  Three 
subjects  responded  that  the  "Direct  To"  navigation  feature  was  very  usable  enroute  and  faster 
with  which  to  navigate.  One  subject  replied  that  the  brightness  of  the  display  was 
advantageous. 


Question  3  asked  subjects  to  list  the  single  worst  feature  of  the  GPS/LORAN.  One  subject 
responded  that  the  time  required  to  .scroll  a  waypoint  was  a  problem.  Two  subjects  replied 
that  signal  strength  in  precipitation  could  be  a  problem. 
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Question  4  asked  subjects  if  they  experienced  difficulty  with  information  retrieval.  One 
subject  replied  that  a  larger,  better  spaced-display  would  help.  One  subject  responded  that 
several  waypoints  were  not  in  the  database  as  shown  on  the  experimental  flight  plans.  Three 
subjects  responded  that  no  difficulties  were  experienced  with  information  retrieval. 

Four  of  five  subjects  replied  that  the  programming  logic  corresponded  with  the  way  that  they 
think  about  and  follow  a  flight  plan  (Question  5).  One  subject  was  unable  to  evaluate  this 
question  because  the  GPS/LORAN  was  not  used  for  actual  course  guidance. 

Question  6  asked  if  subjects  usually  follow  a  flight  plan  or  if  navigation  is  accomplished  by 
flying  waypoint  to  waypoint.  Two  subjects  reported  flying  waypoint  to  waypoint  and  one 
subject  replied  tliat  a  flight  plan  was  usually  followed.  Two  subjects  responded  that  they  do 
not  usually  fly  with  a  GPS/LORAN. 

Question  7  asked  subjects  to  rank  order  the  navigation  information  that  should  be  displayed  at 
all  times.  The  specific  rank  order  of  responses  is  provided  in  Appendix  G.  The  most 
frequently  requested  information  was  time  and  distance  to  a  waypoint  (n  =  5),  followed  by 
the  display  of  crosstrack  error  (n  =  3),  and  the  current  waypoint  navigating  to  (n  =  3).  The 
need  for  waypoint  alert  information,  ground  speed,  course  to  fly,  and  an  indication  that  the 
receiver  is  working  were  each  reported  once. 

Question  8  asked  subjects  to  provide  design  recommendations  according  to  the  knobs/keys, 
labeling,  displays,  and  procedures  of  the  GPS/LORAN.  Two  of  the  five  subjects  provided 
input.  With  respect  to  knobs  and  keys  it  was  suggested  that  large  keys,  that  "beep"  when 
pressed,  would  be  advantageous.  One  subject  replied  that  more  characters  should  be  skipped 
as  the  knob  is  turned  faster  during  scrolling  operations.  With  respect  to  labeling,  larger  and 
better  lighted  labels  were  recommended.  Both  subjects  recommended  a  larger  display.  One 
subject  replied  that  the  LED  "dots"  tended  to  smear  and  suggested  the  use  of  an  LCD  or 
plasma  display.  One  subject  suggested  that  a  space  was  needed  between  the  five  character 
identifiers  and  the  word  "ACK?"  of  the  .secondary  display.  No  procedural  changes  were 
suggested. 


3.6  GENERAL  COMMENTS 

In  addition  to  the  post-experiment  questionnaire,  subjects’  general  comments  were  recorded 
during  each  session.  These  comments  were  grouped  according  to  three  categories:  1 ) 
GPS/LORAN,  2)  SIMULATOR/APPARATUS,  and  3)  EXPERIMENTAL  TASK.  The 
specific  comments  are  provided  in  Appendix  I,  a  summary  is  provided  below. 

Comments  grouped  in  the  GPS/LORAN  category  concerned  the  knobs/keys,  system  logic,  and 
display  characteristics  of  the  receiver.  The  most  frequent  response  was  that  the  waypoint 
"CDI#!"  was  confusing  because  NDB’s  usually  contain  only  three  characters  (n=2).  This 
waypoint  was  viewed  by  all  subjects  and  was  a  valid  database  entry.  The  GPS/LORAN  was 
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rated  as  confusing  because  there  were  many  functions,  and  because  of  the  primary  and 
secondary  displays/controls  delineation.  The  keys  were  viewed  as  being  too  small  and 
difficult  to  press.  The  position  of  the  GPS/LORAN  was  viewed  as  being  potentially 
problematic  during  turbulence  conditions  of  real  llight  because  the  display  was  small. 

Comments  grouped  in  the  SIML'LATOR.  APPARATUS  category  focused  mainly  on  the  flight 
simulation  dynamics.  Nine  responses  were  made  that  portrayed  the  simulatiiMi  dynamics  as 
poor.  Seven  of  these  respon.ses  were  made  during  the  training  session  and  two  were  made 
during  Test  Day  I.  One  comment  that  the  simulator  "feels  closer  to  full-scale  real"  was  made 
during  the  first  test  session.  Two  subjects  reported  that  the  position  of  the  GPS  LORAN 
caused  arm/shoulder  fatigue. 

Comments  grouped  in  the  EXPERIMENTAL  TASK  category  focused  mainly  on  the  demands 
of  the  nighLprogram  conditions.  Three  subjects  rated  the  llighlprogram  task  as  dilficult  to 
perform.  One  subject  developed  a  strategy  to  cope  with  the  simulation  dynamics  in  which  the 
programming  task  was  entered  as  a  series  of  sub-tasks  rather  than  one  large  task.  One  subject 
reported  a  general  strategy  in  which  different  waypoint  entry  methods  were  used  depending 
upon  the  waypoint  category.  For  NDB  and  INT  waypoints,  cursor  entry  was  used  and 
scrolling  was  used  for  the  remaining  categories  because  fewer  APT.  VOR.  and  L’SER 
categories  existed  in  the  database. 
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4.  DISCUSSION 


4.1  PILOT  FLIGHT  AND  GPSA.ORAN  EXPERIENCE 

The  subjects'  flight  experience  data  indicate  that  subjects  had  a  wide  range  (300  to  21,(X)0 
hours)  of  overall  flight  experience.  Subjects’  also  had  a  wide  range  of  total  IFR  time  (60  to 
700  hours)  indicating  that  relatively  low  and  high  IFR  experience  pilots  participated  in  the 
study.  Most  subjects  gained  their  experience  in  a  general  aviation  environment  flying  many 
types  of  single-engine  aircraft.  Overall,  the  subjects  of  the  present  study  had  a  good  amount 
of  general  aviation  experience  and  were  fairly  experienced  under  IFR  flight  conditions. 

Three  of  the  five  subjects  had  prior  experience  with  a  variety  of  LORAN  receivers.  Previous 
LORAN  use  during  flight  was  moderate  with  a  median  estimate  of  45%.  The  display 
readability  and  the  overall  usability  of  all  receivers  were  rated  as  no  worse  than  "easy."  This 
indicates  that  subjects  were  satisfied  with  their  previous  LORAN  receiver  experience.  Right 
plan  creation  and  editing  for  one  receiver  were  rated  as  "somewhat  difficult";  these  functions 
for  ail  other  receivers  were  rated  as  "easy."  Two  subjects  rated  error  recovery  as  "very  easy"; 
one  subject  rated  it  as  being  "neutral."  Controls  operation  ratings  for  all  receivers  ranged 
from  "easy"  to  "neutral." 

Only  one  subject  reported  that  receiver  usage  caused  a  brief  distraction  from  the  primary 
flight  duties.  The  "Direct-To- Waypoint"  procedure  of  one  receiver  was  described  as  difficult 
to  remember.  Suggestions  for  improvements  of  one  receiver  focused  on  the  need  for  better 
operations  and  training  documentation.  This  suggestion  supports  concerns  that  have  been 
raised  by  pilots.  Previous  studies  indicate  that  better  GPS/LORAN  documentation  is  strongly 
needed  for  pilots  and  flight  instructors  (Adams  et  al.,  1 992).  These  authors  report  that  pilots 
rated  operations  manuals  and  training  materials  as  being  poorly  organized  for  effective  usage. 
Furthermore,  training  materials  were  described  as  lacking  in  that  they  do  not  convey  a  good 
understanding  of  system  function. 


4.2  FLIGHT  PERFORMANCE 

In  general,  crosstrack  performance  was  good  for  all  single  task  conditions  (see  Figure  5). 
Single  task  RXTK  performance  improved  for  the  low  and  high  turbulence  conditions  across 
session.  The  best  single  task  performance  occurred  during  the  warm-up  trials  of  the  second 
test  session.  These  data  indicate  that  subjects  maintained  their  manual  flight  skills  after  a  test 
interval  of  at  least  seven  days.  The  effect  of  programming  on  flight  performance  is  shown  by 
the  increase  in  crosstrack  error  for  all  dual  task  (flight/program)  trials  relative  to  single  task 
performance.  Although  performance  was  degraded  for  these  trials,  the  variability  in 
performance  during  training  diminished  by  the  end  of  the  first  test  session.  Performance 
across  test  session  was  maintained  after  a  test  interval  of  seven  days,  indicating  that  subjects 
were  performing  the  flight  task  as  instructed. 


Altitude  deviation  perfomumce  showed  a  trend  similar  to  the  RXTK  scores  (see  Figure  6). 
Single  task  RADV  scores  were  maintained  at  less  than  23  feet  of  deviation  for  all  conditions, 
and  appeared  to  be  stable  across  session.  Dual  task  performance  improved  consistently  for 
the  medium  and  high  turbulence  conditions  and  remained  stable  for  the  low  turbulence 
condition  across  session. 

In  summary,  learning  effects  appeared  to  diminish  by  the  end  of  the  first  test  day  for  both 
flight  performance  measures.  Subjects  were  able  to  maintain  altitude  performance  within  the 
instructed  range  of  +/-  100  feet  for  the  single  and  dual  flight  task  conditions.  Programming 
the  GPS/LORAN  caused  flight  performance  to  degrade  for  all  conditions. 


4.3  ERROR-FREE  GPS/LORAN  PROGRAMMING  PERFORMANCE 

Subjects  were  in  general,  very  accurate  at  the  programming  task.  Eighty-seven  percent  of  the 
trials  were  performed  without  an  error.  In  general,  subjects  required  more  time  to  complete 
the  programming  task  as  turbulence  increased  (see  Figure  7).  suggesting  that  the  flight  task 
was  sufficiently  demanding.  A  slight  test  interval  effect  was  found  for  overall  programming 
time  in  which  more  time  was  required  at  the  high  turbulence  level  of  the  second  day.  There 
were  no  appreciable  differences  for  turbulence  or  test  interval  when  programming  time  was 
examined  by  task  phase  (.see  Table  4).  A  iicnd  was  found  that  suggested  subjects  were 
slightly  quicker  at  the  accuracy  checking  task  during  Test  Day  2. 

The  high  accuracy  rate  and  lack  of  a  large  test  interval  effect  shows  that  subjects  were  trained 
well  and  maintained  their  procedural  skills  over  at  least  a  seven  day  period.  In  addition,  this 
suggests  that  the  programming  task  may  have  been  too  easy. 


4.4  GPS/LORAN  PROGRAMMING  ERRORS 

Errors  in  programming  performance  occurred  on  only  13%  of  the  trials  (see  Table  5). 
Although  the  number  of  errors  committed  was  small,  they  were  apparently  effects  of 
turbulence  and  task.  For  turbulence,  the  number  of  errors  increased  as  turbulence  increased 
from  low  (n=3)  to  medium  (n=6)  to  high  (ri=7).  This  suggests  that  less  attention  may  have 
been  available  for  the  programming  task  at  the  higher  turbulence  levels.  With  respect  to  task 
phase,  more  than  twice  as  many  errors  occurred  during  flight  plan  entry  (ri=l  1)  compared 
with  flight  plan  editing  (n=5).  This  suggests  that  flight  plan  entry  may  have  been  more 
difficult  than  the  flight  plan  editing.  One  explanation  for  this  finding  is  that  the  entry  phase 
required  the  entry  of  four  waypoints  while  the  editing  phase  only  required  two  waypoint 
entries  (see  Figure  4).  In  addition,  a  single  key  press  was  required  to  complete  the  delete 
portion  of  the  editing  phase  once  the  waypoint  was  positioned  on  the  primary  display. 

The  lack  of  a  test  interval  effect  could  be  due  to  two  factors.  It  was  speculated  that  the 
programming  task  may  have  been  too  easy  because  the  flight  task  did  not  require  the  use  of 
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the  GPS/LORAN  receiver.  In  addition,  two  thirds  of  the  training  trials  involved  the 
programming  task,  suggesting  that  subjects  were  highly  trained.  Therefore,  any  test  interval 
effects  may  have  been  masked  by  the  current  experimental  design.  While  this  may  be  the 
case,  the  specific  errors  and  recovery  processes  that  were  observed  do  provide  useful  and 
relevant  information  concerning  the  programming  logic  and  interface  attributes  of  the 
GPS/LORAN  receiver. 

Only  the  Misspell  Waypoint  error  did  not  involve  a  recovery  process.  In  this  case,  an 
incorrect  waypoint  was  entered  into  the  flight  plan  and  was  not  corrected  because  the  error 
was  not  noticed.  In  part,  the  error  may  not  have  been  detected  because  the  incorrect 
waypoint  (JIMMI)  was  phonetically  similar  to  the  desired  waypoint  (JIMIE).  Secondly,  the 
receiver  logic  accepted  "JIMMI"  into  the  flight  plan  because  it  was  contained  in  the  database. 
Lastly,  the  waypoint  was  deleted  from  the  flight  plan  as  part  of  the  editing  portion  of  the 
programming  task.  Because  of  these  factors,  subjects  were  unaware  of  the  error  and  did  not 
initiate  a  recovery.  This  presents  an  unique  problem  in  that,  both  the  operator  and  the 
system  were  functioning  as  if  there  was  no  problem.  The  system  accepted  the  waypoint 
because  it  was  valid  according  to  the  database  and  the  pilot  thought  that  a  correct  waypoint 
was  entered.  It  is  likely  that  this  error  might  have  been  noticed  only  after  flying  off-course 
toward  the  incorrect  waypoint.  This  is  an  example  of  a  system  that  lacks  a  "reality  check" 
for  input  data,  and  has  the  potential  to  seriously  degrade  flight  performance  and  safety 
(Adams  et  al.,  1992;  Wiener,  1985). 

The  remaining  three  error  types  were  recognized  by  subjects  and  involved  a  recovery 
process.  The  ease  or  difficulty  of  recovery  depended  upon  the  "depth"  of  the  programming 
sequence  at  which  the  error  occurred.  Errors  that  were  recognized  early  in  the  programming 
process  were  easier  to  recover  from  than  errors  that  occurred  "deeper"  in  the  process.  A 
second  distinguishing  characteristic  was  whether  or  not  the  error  involved  the  manipulation  of 
the  acknowledge  (ACK)  key.  The  Select  Incorrect  Waypoint  Category  error  did  not  involve 
manipulation  of  the  ACK  key  (compare  Figures  9,  10,  and  11)  and  subjects  recovered 
quickly  and  easily.  The  recovery  method  involved  exiting  and  entering  flight  plan  mode  by 
turning  the  large,  primary  control  knob.  Subjects  were  able  to  continue  the  programming 
process,  relatively  unaffected  by  the  error. 

The  two  remaining  error  types  involved  activation  of  the  ACK  key  at  some  point  during  the 
programming  process.  The  specific  point  at  which  ACK  was  pressed  determined  the  type 
and  severity  of  the  error  and  appeared  to  impact  recovery  as  well.  In  general,  these  errors 
were  more  difficult  to  recover  from  and  required  long  recovery  times.  One  of  these  errors 
was  the  Default  Waypoint  Entry  error. 

The  Default  Waypoint  Entry  errors  occurred  when  ACK  was  pressed  after  a  waypoint 
category  was  selected  but  prior  to  entry  of  an  identifier.  Because  an  identifier  was  not 
entered  prior  to  pressing  the  ACK  key,  the  system  logic  selected  the  default  waypoint  of  the 
current  category.  In  only  one  case,  recovery  from  this  error  was  initiated  immediately  and 
the  subject  deleted  the  unwanted  waypoint  from  the  flight  plan  (see  steps  1-5  and  10  of 
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Figure  10).  The  remaining  four  occurrences  involved  multiple  ACK  presses  and  random 
control  inputs  before  subjects  successfully  deleted  the  incorrect  waypoint  (see  step«  1-4  and 
6-10  of  Figure  10). 

Why  do  subjects  press  the  ACK  key  at  the  improper  time  in  the  programming  sequence  and 
enter  the  default  waypoint?  One  contributing  factor  is  the  multifunction  characteristic  of  the 
ACK  key.  This  key  is  used  for  many  purposes:  1)  To  begin  the  flight  plan  process,  2)  to 
enter  waypoints  into  a  flight  plan,  3)  to  advance  to  the  next  position  in  a  flight  plan  in  order 
to  enter  another  waypoint,  and  4)  to  accept  editing  functions.  As  a  result,  subjects  become 
accustomed  to  pressing  the  ACK  key  any  time  it  flashes  without  determining  the 
consequences  beforehand. 

A  second  factor  is  that  the  status  of  the  ACK  key  does  not  change  once  a  waypoint  has  been 
entered-it  continues  to  flash.  This  design  characteristic  was  problematic  because  subjects 
could  not  remember  if  ACK  wgs  pressed  or  needed  to  be  pressed  in  order  to  proceed  with 
programming.  This  was  evident  in  subjects  behavior  after  switching  attention  to  the 
programming  task  from  the  flight  task;  the  tendency  was  to  press  ACK  upon  returning 
attention  to  the  receiver.  This  is  clearly  an  example  of  a  design-induced  error  that  should  be 
addressed  in  future  receiver  designs. 

Recovery  from  the  Default  Waypoint  Entry  error  was  affected  by  the  layout  of  the  editing 
functions.  In  order  to  reach  the  editing  functions,  the  large,  secondary  knob  must  be  turned 
until  the  insert,  delete,  change,  or  erase  screens  are  shown.  If  the  secondary  knob  was 
turned  in  the  appropriate  direction,  these  functions  were  shown  in  a  serial  order.  If  the  knob 
was  turned  in  the  opposite  direction,  four  screens  of  unrelated  information  (e.g.,  TRSA/ 
ARSA  notices,  distance  and  bearing  to  waypoint)  are  shown.  This  unrelated  information  is 
located  within  the  first  two  detents  of  the  waypoint  entry  screens  and  subjects  would  often 
turn  the  knob  in  this  direction.  Upon  seeing  this  information  subjects  had  a  tendency  to 
become  confused  and  would  not  turn  the  knob  in  the  opposite  (i.e.,  appropriate)  direction. 
Stated  another  way,  subjects  did  not  realize  the  editing  screens  were  within  three  or  four 
detents  of  their  current  position  and  were  not  sure  where  to  find  them.  This  caused 
confusion  and  lengthened  the  recovery  process. 

The  Acknowledge  Invalid  Waypoint  error  is  different  from  the  Default  Waypoint  Entry  error 
in  that  the  identifier  entry  process  was  started  before  ACK  was  pressed  (see  Figure  11).  If 
the  category  or  identifier  name  was  incorrect  and  the  waypoint  was  not  contained  in  the 
database,  pressing  ACK  produced  a  "Saved  Position”  message.  Because  the  waypoint  is  not 
in  the  database,  the  ACK  key  did  not  flash  and  the  "ACK?"  message  on  the  secondary 
display  did  not  appear.  Once  the  "Saved  Position"  message  appears,  the  ACK  key  is 
illuminated  (but  does  not  flash)  until  the  "Saved  Position"  message  was  cleared.  The  only 
method  of  recovery  was  to  press  ACK  once.  This  action  returned  the  screen  to  the  previous 
mode  but  did  not  return  the  user  to  the  exact  previous  state  (i.e.,  previous,  incomplete 
waypoints  are  deleted). 
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Because  this  error  can  only  occur  when  ACK  is  truly  pressed  at  "the  wrong  time"  (i.e.,  not 
flashing),  it  is  likely  that  subjects  pressed  ACK  out  of  habit.  As  described  earlier,  subjects 
were  accustomed  to  pressing  ACK  due  to  the  multifunction  nature  of  the  key  (at  least  two 
presses  are  required  for  each  waypoint  entry).  As  a  result,  they  may  simply  press  ACK 
when  in  doubt  or  simply  due  to  habit.  The  primary  source  of  difficulty  associated  with  this 
error  was  the  inability  to  initiate  a  recovery  sequence  in  a  timely  manner. 

Essentially,  subjects  were  confused  about  the  "Saved  Position"  message  and  were  unsure  of 
how  to  proceed  when  it  was  displayed.  Because  the  ACK  key  remained  illuminated  after  the 
error  occurred,  subjects  were  not  sure  if  it  should  be  pressed  again.  The  steady,  lighted 
ACK  key  was  a  condition  that  subjects  had  only  encountered  during  the  initial  training  (see 
Appendix  C,  p.  C-6).  To  further  complicate  matters,  subjects  were  reluctant  to  press  ACK 
in  order  to  clear  the  message  although  this  was  the  only  recovery  method.  Turning  any  knob 
(except  the  power  knob)  or  pushing  any  key  (except  ACK)  did  not  produce  a  response  in  the 
receiver,  and  subjects  may  have  perceived  that  the  system  was  "locked-up."  This  system 
behavior  combined  with  the  uncertainty  caused  by  the  steady  illuminated  ACK  key,  served  to 
confuse  subjects  and  lengthen  the  recovery  process.  Evidence  of  this  behavior  is  found  in 
that  all  recoveries  involved  random  control  manipulation.  In  one  case,  a  subject  became  so 
frustrated  that  receiver  power  was  turned  "off  in  order  to  reset  the  receiver. 


4.4.1  IMPLICATIONS  FOR  TRAINING 

The  random  behavior  observed  during  error  recovery  suggests  that  formal  training  materials 
(e.g.,  operations  manuals)  should  include  explicit  error  recognition  and  recovery  information. 
In  general,  GPS/LORAN  operation  manuals  do  not  include  information  about  errors  or 
specific  methods  of  recovery.  It  is  also  likely  that  pilots  are  not  trained  by  instructors  in 
how  to  recognize  and/or  recover  from  errors.  As  a  result,  it  is  possible  that  the  confusion 
and  multiple  errors  that  occurred  in  the  laboratory  may  occur  in  actual  flight.  Clearly,  some 
form  of  initial  training  along  with  the  appropriate  documentation  could  in  part,  reduce  error 
recognition  and  recovery  problems. 


4.5  POST-EXPERIMENT  QUESTIONNAIRE 

One  of  the  main  purposes  of  the  post-experiment  questionnaire  was  to  capture  subjects’ 
concerns  about  the  GPS/LORAN  receiver.  Of  all  comments  that  were  received,  only  four 
were  directly  related  to  the  errors  that  occurred.  The  low  number  of  comments  related  to 
errors  suggests  that  subjects  were  unaware  of  why  the  errors  occurred.  In  addition,  subjects 
may  also  have  been  unaware  that  system  logic,  prompting,  and  other  system  attributes  can 
contribute  to  performance  decrements. 

Four  of  five  subjects  replied  that  the  program  logic  corresponded  with  the  way  that  they 
think  about  and  follow  a  flight  plan.  The  low  number  of  errors  supports  this  finding. 
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however,  subjects  showed  high  levels  of  confusion  and  frustration  when  errors  occurred. 

This  dissociation  between  performance  and  subjective  ratings  provides  support  for  the  notion 
that  subjects  were  unaware  that  receiver  design  can  adversely  affect  performance.  This 
finding  also  provides  support  for  the  use  of  botli  objective  and  subjective  measures  to 
evaluate  pilot-receiver  performance  issues  (see  Yeli  &.  W’lckens,  1988). 

One  subject  commented  that  it  seemed  easier  to  misidentify  waypoints  during  editing.  Part 
of  the  confusion  may  be  due  to  the  layout  of  the  flight  plan  instructions.  More  text  was 
required  for  the  editing  instructions  compared  with  the  entry  instructions  (see  Appendix  B). 
This  additional  text  may  have  been  more  difficult  to  interpret  and  could  have  caused 
confusion  among  the  waypoints.  In  addition,  the  organization  of  the  editing  screens  may 
have  contributed  to  this  comment.  Confusion  during  recovery  from  the  Default  Waypoint 
Entry  errors  occurred  when  subjects  viewed  unrelated  information  (e.g.,  TRSA/ARSA 
notices,  distance  and  bearing  to  waypoint)  while  searching  for  the  editing  functions.  Because 
unrelated  information  was  present  at  the  same  level  as  the  editing  functions,  subjects  were 
unsure  of  their  location  within  the  system  structure  and  were  likely  to  become  lost. 

One  subject  replied  that  the  time  required  to  scroll  a  waypoint  was  a  problem.  A  related 
comment  suggested  that  the  amount  of  data  that  is  skipped  should  be  a  function  of  knob 
speed.  This  explains  why  most  subjects  used  the  faster,  "cursor"  method  for  entering 
waypoints.  The  problem  with  this  approach  is  that  it  allows  for  the  occurrence  of  Invalid 
Acknowledge  Waypoint  errors.  That  is,  invalid  waypoint  identifiers  can  only  be  entered  on 
the  secondary  display  using  the  cursor  method.  If  scrolling  is  used,  only  valid  waypoints  are 
displayed  on  the  screen  and  it  is  impossible  to  enter  an  invalid  waypoint. 

It  was  commented  that  several  waypoints  were  not  contained  in  the  database  as  given  in  the 
flight  plans.  Actually,  the  wayfioint  identifiers  were  spelled  correctly  but  the  wrong 
categories  were  chosen.  As  a  result.  Acknowledge  Invalid  Waypoint  errors  occurred  and 
subjects  were  unaware  of  the  reason  why.  There  are  two  factors  that  contribute  to  the 
difficulty  of  recognizing  and  verifying  waypoint  categories.  Subjects’  hands  often  blocked 
the  view  of  the  category  designator  and  subjects  were  often  required  to  release  the  secondary 
knob  to  visually  inspect  the  category.  In  addition,  the  category  designator  was  only 
displayed  on  the  primary  display  after  a  waypoint  was  entered.  At  this  point,  it  was  too  late 
to  simply  change  the  category— the  waypoint  must  be  changed  or  deleted  and  re-entered  into 
the  flight  plan. 

One  comment  that  did  not  appear  to  relate  to  any  specific  error  but  may  have  caused 
confusion  referred  to  the  lack  of  space  between  the  five  character  identifiers  and  the  word 
"ACK?"  on  the  secondary  display.  This  is  purely  a  software  issue  in  that  the  display  space 
exists,  but  the  design  failed  to  take  advantage  of  it.  Other  comments  that  might  have  been 
indirectly  related  to  poor  performance  included  the  need  for:  larger  and  more  generously 
spaced  display(s),  better  layout  of  the  display(s),  larger  keys,  keys  that  "beep"  when  pressed, 
and  larger  and  brighter  lighted  labels. 
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4.6  GENERAL  COMMENTS 


One  subject  replied  that  the  GPS/LORAN  functions  were  confusing  because  there  were  many 
available.  In  addition,  the  primary/secondary  organization  of  controls  and  displays  also 
added  to  confusion  about  system  operation.  A  related  concern  was  that  there  were  many 
different  ways  to  perform  the  same  function.  This  was  perceived  to  produce  confusion  about 
remembering  specific  procedures.  The  complexity  of  GPS/LORAN  receivers  has  been  a 
strong  concern  among  users,  particularly  under  periods  of  high  workload  (Adams  et  al., 
1992).  These  comments  support  previous  findings  that  point  to  non-intuitive  and  complex 
system  structures  as  the  primary  cause  for  errors.  Because  these  systems  provide  many  ways 
of  attaining  the  same  goal,  users  may  not  formulate  an  appropriate  mental  model  of  the 
system.  Hence,  when  trouble  arises,  pilots  may  be  unsure  about  the  proper  recourse, 

A  majority  of  the  comments  related  to  the  simulation  focused  on  the  flight  dynamics.  Seven 
comments  about  the  instability  and  unresponsiveness  of  the  flight  simulation  were  made 
during  the  training  session.  During  the  first  test  session,  one  comment  echoed  these 
concerns  and  another  claimed  that  the  simulation  was  operating  more  closely  to  real  flight 
conditions.  During  the  second  test  session,  one  subject  replied  that  the  simulation  was 
responding  too  slowly.  The  decline  in  the  frequency  of  these  comments  across  session 
suggests  that  subjects  learned  how  to  more  effectively  interpret  and  respond  to  the  flight 
simulation. 

Comments  about  the  experimental  conditions  show  that  subjects  perceived  a  great  deal  of 
demand  associated  with  the  flight/program  dual  task  trials.  Two  subjects  reported  that 
specific  strategies  were  developed  to  compensate.  To  assist  with  time-sharing,  one  subject 
reported  that  it  was  easier  to  program/edit  the  GPS/LORAN  as  a  series  of  sub-tasks. 

Another  subject  used  different  waypoint  entry  methods  depending  upon  the  type  of  waypoint 
category  that  was  required.  This  subject  reported  that  it  was  faster  to  scroll  airports  and 
VOR’s  because  fewer  identifiers  are  in  the  database  compared  with  NDB’s  and  INT’s.  The 
cursor  method  was  used  to  enter  NDB  or  INT  waypoints.  The  fact  that  strategies  were 
developed  suggests  that  subjects  were  developing  internal  models  of  system  function  that 
correspond  to  the  GPS/LO^N  receiver  logic  and  menu  structure. 


4.7  SUMMARY 

The  flight  performance  data  indicate  that  subjects  performed  the  task  as  instructed  and  were 
able  to  maintain  their  manual  flight  skills  after  a  period  of  at  least  seven  days.  Turbulence 
effected  the  programming  task  as  predicted  in  that  more  time  to  program  was  required  as 
turbulence  was  increased.  In  addition,  more  complex  errors  were  committed  at  the  medium 
and  high  turbulence  levels.  The  lack  of  a  marked  test  interval  effect  could  be  due  to  the  fact 
that  subjects  were  highly  trained  and  the  programming  task  may  have  been  too  easy.  The 
programming  task  involved  a  high  degree  of  procedural  knowledge  and  subjects  were  not 
required  to  use  any  of  the  receiver  functions  in  order  to  fly  the  simulation.  Hence,  subjects 
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treated  the  programming  task  almost  as  a  separate  task  altogether  rather  than  an  embedded 
task  which  could  have  made  the  task  easier  to  perform.  Evidence  of  this  is  given  in  that 
strategies  were  developed  to  enhance  either  the  programming  task  or  the  tlight  task. 

Although  the  programming  task  is  similar  to  enroute  programming  conditions,  it  does  not 
require  subjects  to  integrate  the  GPS/LORAN  functions  with  the  Hight  task  and  might  not 
have  been  sufficiently  demanding. 

The  strategies  developed  by  subjects  suggests  that  mental  models  (e.g.,  Reigeluth  & 

Schwartz,  1987;  Rouse  &  Morris,  1986)  were  developed  by  subjects  in  an  attempt  to  better 
understand  the  system  logic  and  menu  structure.  Difficulties  were  found  with  the  editing 
functions  and  could  be  due  to  the  extraneous  information  that  was  contained  among  the 
editing  commands.  Because  subjects  became  easily  confused  when  unrelated  information  was 
viewed,  it  is  increasingly  important  to  determine  how  menu  structures  affect  performance 
both  within  and  across  GPS/LORAN  receiver  types. 

The  Misspell  Waypoint  error  may  appear  to  be  non-critical,  however,  its  occurrence  during 
actual  flight  and  in  the  presence  of  stressors  (e.g.,  traffic,  weather,  abnormal/emergency 
operations)  could  compromise  flight  safety.  Thus,  current  GPS/LORAN  systems  must  be 
evaluated  according  to  how  well  pilots  can  perceive  errors.  Comments  from  this  study 
indicate  that  pilots  were,  in  most  cases,  unaware  of  incorrect  actions  that  were  performed. 

In  addition,  the  random  behavior  exhibited  during  recovery  indicates  that  pilots  were  also 
unsure  of  how  to  recover  in  an  orderly  fashion. 

The  present  research  indicates  that  errors  can  occur  under  fairly  simplified  flight  conditions. 
Although  few  errors  were  committed  overall,  the  fact  that  they  did  occur  while  a  simple 
navigation  task  was  performed  (straight  and  level  flight)  suggests  the  potential  for  more 
serious  problems  under  actual  flight  conditions.  As  GPS/LORAN  receivers  are  accepted  into 
precision  navigation  tasks  (e.g.,  to  replace  or  augment  ILS)  the  flight  demands  will  be  high 
and  new  problems  in  pilot/receiver  performance  are  likely  to  emerge.  Hence,  receiver 
designs  of  the  next  generation  must  provide  more  intuitive  interface  designs  by  considering 
the  pilots’  ability  to  operate  within  the  menu  structure  and  to  recognize  and  recover  from 
errors  when  mistakes  are  made. 


4.8  FOLLOW-ON  RESEARCH 

Follow-on  research  needs  to  be  conducted  to  validate  the  present  concerns  and  evaluate  the 
design  attributes  of  additional  GPS/LORAN  receivers.  Future  evaluations  should  also 
consider  the  context  in  which  receivers  will  be  used  to  navigate  within  the  National  Airspace 
System.  The  cognitive  demands  associated  with  enroute  and  terminal  operations  are  different 
and  receiver  designs  should  be  evaluated  in  both  contexts.  Terminal  procedures  for  example, 
may  require  that  pilots  choose  one  of  several  pre-determined  approach  procedures  for 
landing.  Will  the  receiver  sequence  the  pilot  through  the  approach  automatically  or  will  the 
pilot  be  able  to  choose  waypoints  within  the  pre-determined  approach?  What  are  the  effects 
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of  air  traffic  control  (ATC)  changes?  Can  pilots  quickly  and  accurately  adjust  the  pre¬ 
determined  jqjproach  sequence  as  needed?  In  addition,  pilots  are  likely  to  be  sequenced  into 
the  2q)proach  path  at  various  points  depending  upon  traffic.  As  a  r^ult,  the  manner  in  which 
a  receiver  allows  the  pilot  to  enter  the  2q;)proach  at  different  points  will  be  critical.  A  related 
concern  is  the  missed  sqiproach  procedure.  Will  the  missed  approach  procedure  (MAP)  be 
pre-programmed  or  will  the  pilot  be  required  to  program  the  MAP  as  part  of  the  ^proach 
duties?  How  will  required  changes  in  the  MAP  be  accommodated  when  weather  or  traffic 
are  encountered? 

The  enroute  duties  also  need  to  be  further  evaluated.  The  present  study  examined  rnily  the 
straight  and  level  flight  duties  required  of  enroute  operations.  In  addition,  the  different 
demands  associated  with  IFR  and  VFR  flight  are  lilrely  to  affect  programming  performance. 
What  are  the  effects  of  ATC  commanded  flight  plan  changes  whUe  enroute?  Can  pilots 
effectively  monitor  radio  communications,  navigate  within  the  required  airspace  corridors, 
avoid  traffic,  and  make  program  changes  simultaneously?  The  current  receiver  designs  must 
be  evaluated  under  the  above  conditions  to  determine  which  design  attributes  facilitate 
performance  and  which  have  the  potential  to  degrade  performance.  The  favorable  design 
attributes  can  be  used  to  influence  the  next  generation  GPS/LORAN  receiver  designs. 


4.9  IMPLICATIONS  FOR  GPS/LORAN  RECEIVER  DESIGN 

The  following  design  recommendations  are  based  on  programming  performance,  subjective 
responses,  experimenter  observations,  and  the  relevant  literature.  The  General 
Recommendations  section  is  related  to  improving  the  overall  usability  of  GPS/LORAN 
receivers.  The  following  sections  address  specific  controls,  functions,  system  logic,  error 
recognition,  and  error  recovery  issues. 


4.9.1  GENERAL  RECOMMENDATIONS 

Perhaps  the  simplest  and  most  overlooked  design  attribute  is  the  need  for  a  backspace, 
escape,  or  clear  key.  Whatever  the  function,  pilots  must  have  the  ability  to  correct  data 
entry  mistakes  quickly  and  easily.  The  use  of  a  backspace,  escape,  or  clear  key  could 
facilitate  performance  by  reducing  "restarts"  in  programming  because  of  simple  data  entry 
mistakes. 

Some  pilots  stated  that  display  character  spacing  was  problematic.  Mangold,  Eldredge,  and 
Lauber  (1992)  suggest  a  minimum  separation  of  one  character  space  between  words  or 
distinct  display  segments.  A  related  concern  is  the  minimum  character  height  required  for 
text  readability.  Characters  and  labels  subtending  at  least  30  minutes  of  visual  angle  for 
critical  information  and  20  minutes  of  visual  angle  for  non-critical  information  is  suggested 
(Mangold  et  al.,  1992).  At  the  viewing  distance  of  the  present  study,  minimum  character 
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heights  of  0.23  inches  (critical  information)  and  0.  IS  inches  (non-critical  information)  are 
suggested.  These  measurements  require  validatitm  under  turbulence  arul  vibration  conditions. 


Because  lighting  conditions  vary  between  extremes  during  flight,  it  is  desirable  to  provide 
user  contrc  'able  lighting.  Mangold  et  al.  (1992)  report  contrast  rsUios  of  10:1  for  indirect 
ambient  lighting  and  20: 1  for  direct  ambient  lighting  conditions. 

Because  pilots  may  not  always  be  able  to  view  an  actual  key  press,  it  would  be  advantageous 
to  provide  confirmation  of  key  presses.  One  pilot  suggested  that  the  keys  ”beq)"  when 
pressed  to  provide  feedback;  McCormick  and  Sanders  (1982)  suggest  audible  "clicks.”  As 
McAnulty  0993)  points  out,  auditory  signals  may  not  be  the  best  choice  in  a  coclq)it 
environment  because  of  high  ambient  noise  levels.  As  an  alternative,  tactile  feedback  can  be 
used  to  provide  information  that  a  key  has  been  pressed.  Tactile  feedback  may  help  support 
time-sharing  because  it  does  not  increase  the  visual  demands  associated  with  flight. 

Scrolling  speed  of  characters  was  mentioned  as  a  concern  by  one  pilot.  It  seems  intuitive 
that  character  scrolling  speed  should  ramp-up  (to  some  limit)  as  knob  speed  increases  to 
reduce  the  amount  of  time  required  to  enter  waypoints.  An  algorithm  of  this  type  should 
allow  users  to  input  large  changes  with  large  knob  turns  and  "tune-in"  closer  and  closer  to 
the  desired  target  with  smaller  knob  turns.  A  related  concern  is  maintaining  consistent  and 
efficient  control  manipulations.  For  example,  continuous  knob  turns  should  cause  the  entire 
alphanumeric  database  to  repeat  as  the  last  character  is  passed.  Some  systems  contain  a 
"stop"  point,  in  which  the  user  must  scroll  in  the  reverse  direction  to  reach  the  beginning  of 
the  database.  This  design  requires  large  amounts  of  knob  turning  and  is  not  an  efficient 
design  choice. 

Additionally,  key  sizes  were  remarked  as  being  too  small  and  difficult  to  operate. 
McCormick  and  Sanders  (1982)  suggest  a  minimum  key  size  of  0.5  inches  and  at  least  0.5 
inches  of  separation  between  keys  for  fingertip  operation.  McAnulty  (1993)  reports  that 
minimum  key  sizes  between  0.385  and  0.50  inches,  with  a  minimum  separation  of  0.25 
inches,  should  suffice. 


4.9.2  DEDICATED  FUNCTIONS 

A  dedicated  function  is  defined  as  an  action  that  is  required  to  be  available  at  all  times 
(regardless  of  the  current  mode),  and  may  also  require  a  single  key  press/knob  turn  to 
activate.  This  type  of  function  should  have  a  distinct  control  that  is  not  shared  by  other 
functions  or  modes.  In  addition,  if  the  function  is  very  easy  to  activate  (e.g.,  a  single  key 
press),  it  should  require  a  confirmation  step  to  reduce  unwanted  activations.  Users  should 
also  have  the  ability  to  "escape"  or  "back-up"  within  a  process  if  an  unwanted  activation 
occurs. 
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4.9.3  MULTIPLE  FUNCTION  CONTROLS 


A  multiple  function  control  is  defiiMd  as  a  control  that  activates  diffoent  fiuictions  dqxnding 
on  the  system  mode.  Multiple  function  controls  are  problematic  because  th^  contribute  to 
confusion,  errors,  and  imqrpropriate  amtrol  manipulations.  To  reduce  the  i^oblems 
associated  with  multiple  function  controls,  users  ^ould  be  clearly  aware  of  what  a  control 
will  do  in  any  given  mode  (Hutchins,  HoUan,  &  Norman,  1986).  One  method  of  achieving 
this  is  to  provide  appropriate  display  prompts.  For  example  consider  key  XYZ:  in  one  case 
this  key  accepts  a  waypoint  into  a  flight  plan,  while  in  another  case  it  advances  the  active 
position  within  a  flight  plan.  Display  prompts  such  as  "Press  XYZ  to  add  waypoint”  and 
"Press  XYZ  to  advance,"  respectively,  could  reduce  the  likelihood  of  confusion  and  control 
errors.  Although  the  specific  content  and  Iragth  of  messages  would  need  to  be  determined 
and  evaluated,  the  idea  is  to  provide  the  user  with  unambiguous,  easy  to  interpret  informatimi 
about  the  effects  of  a  control  manipulation  for  each  system  mode. 


4.9.4  SYSTEM  LOGIC  AND  MENU  STRUCTURE 

System  logic  and  structure  issues  refer  to  the  organization  of  the  various  menus,  pages,  or 
screens  of  display  information  that  are  contained  within  a  particular  mode.  Although 
research  is  required  to  determine  the  exact  structure  and  layout  for  the  specific  system 
modes,  some  generalizations  can  be  made.  Information  that  is  unrelated  to  the  purpose  of 
the  mode  (i.e.,  the  task)  should  not  be  contained  within  that  mode  because  of  the  potential 
for  confusion.  For  example,  TRSA  and  ARSA  information  should  not  be  included  at  the 
same  level  as  flight  plan  editing  commands.  Modes  should  also  contain  instructional  prompts 
that  convey  the  state  of  the  mode  to  the  user.  This  feature  could  reduce  errors  associated 
with  the  memory  and  divided  attention  limitation  of  human  operators  (Miyata  &  Norman, 
1986).  For  example,  flight  plan  modes  should  contain  distinct  "begin”  and  "end"  points,  as 
well  as  prompting  to  guide  the  user  through  each  stage  of  the  flight  plan  entry  process  (e.g.. 
Enter  Waypoint?,  Edit  Waypoint?,  and  Flight  Plan  Complete?).  These  attributes  may  make 
it  less  likely  for  pilots  to  forget  or  misinterpret  what  has  and  what  needs  to  be  accomplished 
while  time-sharing  receiver  programming  with  other  flight  duties. 

Research  is  required  to  determine  the  optimal  logic  structures  for  the  various  modes  of 
GPS/LORAN  receivers.  One  approach  is  to  design  the  logic  structures  according  to  the 
duties  for  each  phase  of  flight  (Adams  et  al.,  1992).  This  implies  that  some  modes  may 
function  differently  across  a  particular  flight.  How  will  pilots  respond  to  these  differences? 
What  happens  when  well-learned  procedures  require  "new"  actions?  Can  errors  be  easily 
recognized  and  recovered  from  in  each  mode?  These  concerns  need  to  be  addressed  in  order 
to  evaluate  the  impact  of  designing  by  phase  of  flight. 
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4.9.5  ERROR  RECOGNITION  AND  RECOVERY 


Typically,  error  recognition  and  recovery  is  accomplished  by  comparing  the  intended  result  of 
an  action  with  the  actual  result.  For  this  to  occur,  the  user  must  understand  the  logic  and 
menu  structure  of  the  system,  how  to  operate  the  controls  (multiple  and  dedicated  function 
controls),  and  the  relationship  between  the  controls  and  system  logic  (Lewis  &  Norman, 

1986).  Pilots  in  the  present  study  experienced  difficulty  in  at  least  one  or  all  of  these  areas 
when  an  error  occurred.  The  design  recommendations  made  in  the  above  sections  serve  to 
improve  the  users’  ability  to  evaluate  the  intended  and  expected  actions,  and  therefore 
improve  error  recognition  and  recovery.  In  addition,  it  is  important  to  consider  the  need  for 
providing  redundant  error  recognition  and  recovery  information. 

One  method  of  providing  redundant  error  information  is  to  "alert  the  user"  via  the  display  or 
other  media  that  something  was  done  incorrectly.  Although  it  is  impossible  to  predict  all 
possible  errors,  consideration  of  display  prompts  for  disastrous  errors  would  be  advantageous. 
For  example,  the  user  could  be  alerted  if  the  category  or  identifier  of  a  desired  waypoint  does 
not  exist  in  the  database.  A  message  such  as  "Waypoint  Does  Not  Exist,  Create?"  would 
prompt  the  user  to  check  the  spelling  and  edit  the  waypoint,  or  accept  it  as  is. 

Users  should  also  be  alerted  about  similar  waypoints  that  exist  in  the  same  or  different 
categories.  For  example,  user  waypoints  that  sound  the  same  (e.g.,  DREEM  vs.  DREAM  and 
JIMIE  vs.  JIMMI)  could  easily  be  transposed  within  a  flight  plan.  To  avoid  confusion  the 
user  could  be  given  both  waypoints  and  prompted  to  choose  the  correct  one  (e.g.,  "DREEM 
or  DREAM?").  Waypoints  of  different  categories  that  are  spelled  the  same  (ONA-airport  and 
ONA-VOR)  may  also  be  easily  confused.  An  obvious  factor  is  the  spelling  (some  receivers 
add  a  K  to  the  beginning  of  all  airports  to  avoid  this  problem),  however  a  related  concern  is 
the  physical  layout  of  the  category  selector.  Because  the  airport  and  VOR  categories  are 
more  frequently  used,  they  are  located  adjacently  on  most  receivers.  This  also  affords  the 
accidental  incorrect  selection  of  categories.  Separation  of  the  categories  and  providing  large 
category  identifiers  could  minimize  this  concern. 

Like  the  system  logic  and  menu  structure  issues,  error  recognition  and  recovery  methods  need 
to  be  developed  and  tested.  This  study  shows  that  current  display  and  logic  configurations  do 
not  always  promote  easy  recognition  and  recovery  of  errors.  In  some  cases,  multiple  errors 
resulted  because  pilots  were  unsure  of  what  to  do  or  why  an  error  occurred.  Clearly,  these 
problems  must  be  minimized  to  ensure  the  safe  and  effective  use  of  GPS/LORAN  receivers  in 
precision  navigation  tasks. 
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APPENDIX  A 


FLIGHT  INSTRUMENT  DESCRIPTION 


A-1 


FLIGHT  INSTRUMENT  DESCRIPTION 


Flight  Instruments 

(Clockwise  from  lower  left  corner  of  Figure  2) 

(Primary  instruments  in  bold) 

Automatic  Direction  Finder  (ADF) 

This  instrument  has  a  rotatable  compass  rose  dial  and  a  needle  that  indicates  the 
relative  bearing  of  the  NDB  (non-directional  beacon)  station  from  the  aircraft.  This 
instrument  will  be  inoperative  (INOP)  during  the  experiment. 

Turn  Coordinator 

The  single  hash  marks  on  the  right  and  left  side  indicate  that  the  aircraft  is  in  a 
"standard  rate"  (3°/second  or  360^/two  minutes)  turn.  Displacement  of  the  "ball" 
indicates  an  uncoordinated  turn  that  can  be  corrected  by  applying  rudder  of  the 
same  side  as  the  ball  (i.e.,  step  on  the  ball).  This  instrument  is  operative  (OP) 
during  the  experiment. 

Airspeed  Indicator 

The  speed  of  the  aircraft  is  displayed  as  indicated  airspeed  (IAS)  in  knots  (kts). 

The  indicator  has  three  colored  bands  and  a  red  marker.  The  green  band  is  the 
normal  operational  speed  range,  and  the  white  band  is  the  allowable  flaps 
operation  range,  the  yellow  band  is  a  caution  range,  and  the  red  mark  is  the  never 
exceed  speed.  The  lowest  point  of  the  white  range  is  full  flaps  stall  speed.  This 
instrument  is  OP  during  the  experiment.  Each  trial  will  always  begin  at 
approximately  100  kts  airspeed  and  will  vary  throughout  a  trial  depending  on 
aircraft  attitude.  DO  NOT  ATTEMPT  TO  ADJUST  AIRSPEED  USING  THE 
THROTTLE  CONTROL. 

Outside  View  Display 

This  display  represents  the  out  the  window  view  and  will  remain  gray  during  the 
entire  experiment  to  simulate  instrument  meteorological  conditions  (IMG).  This 
instrument  is  OP  during  the  experiment. 

ADF  Tuner 

The  lower  left  instrument  is  the  ADF  tuner  and  will  be  INOP  for  the  experiment. 


Magnetic  Comoass/Marke^  Beacons 

The  magnetic  compass  is  shown  in  the  upper  portion  of  this  display  and  is  OP 
during  the  experiment.  This  instrument  behaves  like  a  "real"  compass  and 
therefore  is  vulnerable  to  lead,  lag,  dip,  acceleration  ,  and  oscillation  errors  due  to 
flight  attitude;  it  is  best  not  to  rely  on  the  compass  for  heading  information. 
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The  lower  portion  of  this  instrument  displays  the  outer,  middle,  and  inner  marker 
beacon  annunciators.  This  instrument  will  be  INOP  during  the  experiment. 

Navigation  Radio  (NAVI  Cluster 

The  upper  and  lower  instruments  display  the  navigation  aid  frequencies  that  are 
tuned  in.  The  upper  navigation  radio  (NAVI)  is  tuned  to  a  VOR  and  displays 
distance  (DME)  information.  This  instrument  is  preset  and  will  not  require 
adjustment  during  a  trial.  DO  NOT  ATTEMPT  TO  ADJUST  THE  NAVI  RADIO.  The 
NAV2  (lower  unit)  radio  is  INOP  during  the  experiment. 

Tachometer 

This  instrument  shows  the  number  of  engine  revolutions  per  minute  (RPM).  Each 
trial  will  begin  at  approximately  2400  RPM  and  will  vary  according  to  aircraft 
attitude.  DO  NOT  ATTEMPT  TO  ADJUST  THE  RPM's  MANUALLY. 

Fuel  Cluster 

The  left  and  right  fuel  cell  capacities  are  shown  and  will  always  begin  at  full  for 
each  trial.  This  instrument  is  OP  during  the  experiment  and  will  decrease  as  flight 
time  increases. 

Engine  Instruments 

Cylinder  head  temperature  (CHT),  exhaust  gas  temperature  (EGT),  oil  pressure  (OIL 
P),  oil  temperature  (OIL  T),  and  an  ammeter  (AMP)  collectively  display  engine 
status.  All  instruments  are  OP  during  the  experiment. 

Clock 

A  clock  displays  elapsed  time  for  each  trial. 

Throttle  Cluster 

The  three  knobs  indicate  throttle  position  (left  side),  mixture,  and  propeller  settings 
(right  side)  and  will  not  require  adjustment  during  the  experiment.  DO  NOT 
ATTEMPT  TO  ADJUST  THE  THROTTLE  DURING  THE  EXPERIMENT. 

Elevator/Trim/Flao  Position  Indicators 

The  elevators,  trim,  and  flap  settings  are  shown  in  this  display.  The  small  left  hash 
mark  displays  elevator  displacement  (up  or  down).  The  triangle  indicator  shows 
the  elevator  trim  setting,  and  the  right  knob  shows  flap  position.  The  trim  and  flap 
positions  are  preset  and  will  not  require  adjustment  during  the  experiment.  DO 
NOT  ATTEMPT  TO  ADJUST  THE  TRIM  AND  FLAPS  DURING  THE  EXPERIMENT. 

Gvro  Compass/Directional  Gvro 

This  instrument  provides  more  accurate  heading  information  than  the  traditional 
magnetic  compass.  However,  this  instrument  is  vulnerable  to  precession  over  time 
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and  cannot  be  adjusted  to  compensate.  Do  not  use  this  instrument  for  precise 
tracking  information. 

Attitude  Indicator 

Aircraft  pitch  and  bank  are  shown  with  this  instrument.  The  vertical  hash  marks 
are  graduated  in  5°  increments  for  pitch.  The  bank  index  (outer  circle)  is 
graduated  in  10°  increments  up  to  the  first  30°  of  bank,  and  in  30°  increments 
for  up  to  90°  of  bank.  The  blue  portion  of  the  indicator  indicates  the  direction  of 
the  sky  and  the  gray  portion  indicates  the  earth  relative  to  the  aircraft  attitude. 

Altimeter 

This  instrument  shows  aircraft  altitude  (above  sea  level)  and  is  preset  at  5000  feet 
for  each  trial.  The  barometric  pressure  is  preset  at  29.92  and  does  not  require 
adjustment.  DO  NOT  ATTEMPT  TO  ADJUST  THE  ALTIMETER  DURING  THE 
EXPERIMENT. 

VOR1A/OR2 

The  number  one  VOR  (VI)  is  operational  and  the  course  deviation  needle  (GDI 
needle)  should  be  used  for  primary  tracking  information.  The  GDI  needle  will  be 
displaced  toward  the  left  or  right  depending  upon  aircraft  heading  relative  to  the 
tuned  in  course.  DO  NOT  ATTEMPT  TO  ADJUST  THE  VOR  DURING  THE 
EXPERIMENT  TO  GOMPENSATE  FOR  GOURSE  DEVIATIONS,  FLY  THE  AIRGRAFT 
TO  MAINTAIN  A  GENTERED  GDI  NEEDLE.  The  second  VOR  (V2)  is  INOP  during 
the  experiment. 

Vertical  Speed  Indicator 

This  instrument  indicates  the  rate  of  climb  and  descent  in  hundreds  of  feet  per 
minute.  This  instrument  is  OP  during  the  experiment  and  will  vary  according 
aircraft  attitude. 
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SAMPLE  FLIGHT  PLAN 


PROGRAM  SEGMENT 

Flight  Plan  1 : 

Waypoint  Category 
GFL  APT 

PONDS  USER 

SXD  NDB 

EEN  VOR 

EDIT  SEGMENT 

INSERT  SHERY  USER  BETWEEN  GFL  APT  AND  PONDS  USER 

DELETE  PONDS  USER 

CHANGE  SXD  NDV  TO  KHRIS  INT 

ACCURACY  CHECK  SEGMENT 
GFL  APT 

SHERY  USER 

KHRIS  INT 

EEN  VOR 

Note:  Underlined  descriptions  not  included  on  experimental  flight  plans. 
The  accuracy  check  segment  was  printed  on  the  reverse  side  of  the  page. 
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I.  Overview 


The  purposes  of  this  experiment  are  to  determine  how  long  a  person  retains 
the  knowledge  required  to  operate  a  GPS/LORAN  receiver,  to  identify  the 
GPS/LORAN  operations  that  are  most  prone  to  error  and  to  identify  the  types  of 
errors  that  are  made  while  performing  GPS/LORAN  operations.  First  we  will  teach 
you  how  to  operate  a  GPS/LORAN  receiver  and  allow  you  to  practice  using  it. 
Tomorrow  will  be  the  first  session  in  which  you  will  perform  a  series  of  operations 
and  we  will  record  the  time  needed  to  perform  these  operations  as  well  the  number 
and  types  of  errors  you  might  commit.  The  second  session  will  be  in  1  or  2 
weeks. 

The  first  step  is  to  train  you  to  operate  the  GPS/LORAN.  If  you  have  any 
questions  during  the  training,  don't  hesitate  to  ask.  You  may  end  your 
participation  in  the  experiment  at  any  time.  As  you  read  these  instructions,  you 
will  notice  that  examples  of  specific  programming  procedures  will  appear  in  bold 
text. 


II.  Waypoints  and  GPS/LORAN 

Only  if  questions  are  asked: 

First  I  would  like  to  familiarize  you  with  some  of  the  terms  I  will  be  using  to 
explain  GPS/LORAN  navigation.  VORs  are  radio  beacons  that  transmit  beams 
called  radials  outward  in  every  direction.  A  receiver  in  the  airplane  detects  these 
signals  and  indicates  the  radial  on  which  the  aircraft  is  located,  thus  allowing  a 
pilot  to  follow  a  radial  to  or  from  a  VOR.  NDBs  are  radio  beacons  iike  VORs  but 
they  are  nondirectional,  that  is  they  do  not  transmit  radiais.  An  intersection  is  the 
point  where  radials  from  two  or  more  VORs  cross. 

Waypoints  are  points  on  the  ground  specified  by  coordinates  of  latitude  and 
longitude.  They  may  be  airports  or  other  navigational  aids  such  as  VORs,  NDBs, 
radial  intersections  or  other  locations  specified  by  the  user. 

The  GPS/LORAN  is  used  to  1 )  store  the  locations  of  these  waypoints  as  well 
as  their  names,  or  identifiers;  2)  to  store  the  waypoints  in  a  prearranged  order 
called  a  flight  plan;  and  3)  to  provide  navigational  information  to  the  pilot  during 
flight. 


Among  the  operations  that  I  will  teach  you  are:  how  to  create  and  edit  a 
flight  plan,  how  to  create  your  own  waypoints  and  save  them,  and  how  to  view 
navigation  information. 


III.  Database 

This  receiver  has  an  internal  database  of  over  20,000  waypoints.  The 
database  is  divided  into  5  categories:  APT,  NDB,  INT,  VOR,  and  USER.  The  first 
four  are  for  storage  of  airports.  NDBs,  intersections,  and  VORs  respectively.  The 
USER  category  is  for  storing  in  the  database  any  additional  waypoints  that  you 
specify. 

The  database  in  this  receiver  may  contain  more  information  about  a  waypoint 
than  its  name  and  position.  For  example,  if  you  look  at  one  of  the  airport 
waypoints,  the  GPS/LORAN  can  tell  you  the  name  and  city,  the  runway  headings 
and  length,  and  the  various  communication  channels,  as  well  as  the  lat/lon.  For 
VORs  and  NDBs,  the  receiver  can  tell  you  the  city  and  state,  its  name,  frequency, 
latitude,  and  longitude.  The  name  of  a  VOR  or  NDB  is  not  necessarily  the  same  as 
its  identifier.  It  will  also  display  the  latitude  and  longitude  of  an  intersection.  I  will 
show  you  how  to  recall  any  waypoint  from  the  database  so  that  you  can  review 
the  information  on  it. 

1 .  There  are  two  sets  of  knobs,  each  with  a  small  inner  and  large  outer 
knob.  In  general,  the  large  outer  knob  is  used  to  select  the  function  or 
mode  and  the  small  inner  knob  is  used  to  scroll  through  the  data 
available  in  that  function.  You  turn  the  large  left  knob  to  the  category  of 
the  waypoint  that  you  are  interested  in  and  turn  the  large  right  function 
knob  to  the  INFO  setting.  So  lets  say  you  want  to  know  the  lat/lon 
coordinates  of  the  MILIS  intersection. 

2.  Turn  the  large  left  knob  to  the  INT  setting  with  the  large  right  knob  on 
INFO. 

3.  As  you  see,  the  receiver  does  not  immediately  show  MILIS.  To  get  to  it 
you  must  turn  the  small  left  knob  to  scroll  through  the  alphabetically 
arranged  waypoints  in  that  category.  Alternatively,  you  could  get  to  it 
by  the  character  entry  method. 

4.  Press  the  CRSR  button  on  the  side  of  the  display  where  the  waypoin* 
identifier  is  displayed.  The  cursor  button  will  light  up  to  indicate  that 
is  in  the  editing  mode.  The  first  letter  will  begin  to  flash. 
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5.  Using  the  small  left  knob,  change  the  flashing  letter  to  the  first  letter  of 
the  waypoint  you  are  interested  in.  The  characters  are  displayed  in  the 
following  order  by  turning  the  knob  from  left  to  right  from  the  beginning: 
a  space,  the  letters  of  the  alphabet,  numbers  1  through  9  and  some 
other  characters  such  as  slashes  and  number  signs. 

6.  Turn  the  large  left  knob  to  move  the  cursor  over  to  the  next  space 
where  you  select  the  next  letter  by  the  same  method.  Continue  this 
until  the  entire  waypoint  identifier  is  entered.  As  you  scroll  through  the 
letters,  you  will  see  the  information  for  the  waypoints  in  the  database. 

If  there  is  no  information  displayed,  then  there  is  no  waypoint  in  the 
database  with  that  identifier.  Do  not  press  the  CRSR  when  no 
information  is  displayed  because  the  GPS/LORAN  will  think  you  are 
trying  to  enter  a  new  waypoint  with  that  identifier.  For  example  there  is 
no  waypoint  for  PMU  APT  but  there  is  one  for  PMV  APT. 

7.  Press  the  CRSR  button  once  again  to  leave  the  character  entry  mode. 
Turn  the  right  inner  knob  to  scroll  through  the  information  on  that 
waypoint.  As  we  turn  the  knob,  we  can  see  the  latitude  coordinate  for 
MILIS  is  42.06.3  and  the  longitude  coordinate  is  71.18.7. 

Demonstrate  once  again  with  an  APT  waypoint  and  then  allow  them  to 
recall  2  waypoints.  Scroll  through  the  information  on  each,  describe 
each  page  of  information.  Ask  them  to  try  both  the  scrolling  and 
character  entry  methods  of  recalling  the  identifier,  advise  when  to  use 
scrolling. 

There  may  be  times  when  you  do  not  know  the  waypoint  identifier  but  you 
might  know  the  common  name  for  the  waypoint  or  its  city  of  location.  I  will  show 
you  how  to  retrieve  the  waypoint  even  though  you  do  not  know  the  identifier. 

1 .  Turn  to  the  category  of  the  waypoint  you  are  searching  for,  in  this  case 
APT,  using  the  large  left  knob  as  you  did  before,  with  the  right  knob  set 
on  INFO.  (It  does  not  matter  that  the  waypoint  displayed  is  not  the  one 
you  are  searching  for  at  this  point). 

2.  Turn  the  small  right  knob  to  display  either  the  name  or  the  city, 
whichever  you  decide  to  search  for.  If  you  know  the  name  of  the  airport 
but  nothing  else,  turn  to  the  page  displaying  the  name  of  the  airport  that 
appeared  on  the  screen  when  you  turned  to  the  APT  category.  Note: 
Make  certain  that  you  do  not  confuse  the  page  displaying  the  city  of 
location  with  the  page  indicating  the  name  of  the  waypoint.  The  name 
of  the  waypoint  is  not  the  same  thing  as  the  identifier.  The  location 
page  will  appear  first  as  you  scroll  the  information  knob  from  left  to  right 
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and  it  will  have  the  abbreviation  for  the  state  on  the  right  hand  side  of 
the  display.  The  name  page  will  appear  next  but  it  will  not  have  the 
state  abbreviation  on  the  right. 

3.  Press  the  right  CRSR  button  and  enter  the  first  four  or  five  characters  of 
the  desired  name  or  city. 

4.  When  you  find  the  desired  waypoint,  press  CRSR  again  to  turn  the 
cursor  off. 

So  lets  say  you  know  there  is  an  airport  in  Fitchburg  but  you  don't  know  the 
identifier  and  you  want  to  have  information  on  the  airport. 

1 .  Select  the  APT  category.  (APT on  (eft,  INFO  on  right) 

2.  Using  the  small  right  knob,  turn  to  the  location  page  for  the  airport  that 
appears. 

3.  Press  CRSR  to  enter  the  location.  Spell  out  Fitchburg  using  the 
character  entry  method.  You  probably  won't  have  to  spell  out  the  whole 
word  because  the  GPS/LORAN  fills  in  the  rest  of  the  word  if  it  is  in  the 
database. 

4.  Once  Fitchburg  is  on  the  display.  Press  CRSR  to  enter  the  location  and 
then  you  can  view  any  of  the  information  on  the  waypoint  as  you  did 
before,  by  turning  the  small  right  knob. 

Often  a  city  name  will  exist  in  several  different  states.  Check  the  state  code 
to  the  right.  If  it  is  not  the  one  you  are  looking  for,  while  in  character  entry  mode, 
turn  the  small  left  knob  to  scroll  the  waypoints,  checking  the  state  code  to  the 
right  until  you  see  the  one  you're  searching  for.  Waypoints  will  be  cycling 
alphabetically  according  to  identifier. 

Demonstrate  again,  using  a  VOR  located  in  Springfield  MO.  Allow  them  to  recall 
two.  Ask  them  to  try  both  scrolling  and  character  entry  methods. 


IV.  How  to  Create  a  Flight  Plan 

A  flight  plan  is  a  sequence  of  waypoints  that  define  the  route  that  the  pilot 
plans  to  follow.  The  flight  plan  may  consist  simply  of  a  starting  waypoint  and  a 
destination  waypoint,  or  there  may  be  several  waypoints  in  between,  show  or 
draw  a  diagram  The  pilot  flies  from  the  first  waypoint  to  the  second  to  the  third 
and  so  on  until  he  or  she  reaches  the  destination.  A  leg  of  the  flight  plan  is  a 
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course  line  from  one  waypoint  to  another.  Using  the  GPS/LORAN  you  can  store  a 
sequence  of  up  to  twenty  waypoints,  or  nineteen  legs  to  form  a  flight  plan. 

1 .  Turn  to  the  flight  plan  mode,  FPL,  with  the  large  left  knob.  If  there  is  no 
flight  plan  stored,  there  will  be  a  string  of  question  marks  in  the  left  half 
of  the  display.  The  right  half  of  the  display  may  show  "Undefined  Leg" 
in  which  case  you  turn  the  right  outer  knob  until  you  see  "Add  Waypoint 
ACK?." 

2.  You  will  see  the  ACK  button  on  the  far  right  side  begin  to  flash.  The 
receiver  does  this  to  prompt  the  user,  and  you  will  see  it  quite  often 
while  using  the  receiver.  Whenever  the  ACK  button  is  flashing,  the 
receiver  is  asking  you  if  you  would  like  to  acknowledge,  or  carry  out  the 
action  described  on  the  display.  You  will  have  to  press  ACK  to 
acknowledge  the  operation  displayed.  If  the  button  flashes  and  there  is 
no  prompting  question  on  the  display  such  as  "Add  Waypoint  ACK?", 
then  the  receiver  is  trying  to  advise  you  of  a  potentially  dangerous 
condition  or  situation  of  which  you  should  be  aware.  In  that  case  you 
should  press  ACK  to  read  the  message  and  press  ACK  again  to 
acknowledge  that  you  understand  the  advisory.  In  this  case  it  is 
prompting  you  with  the  question  of  whether  or  not  you  would  like  to 
add  a  waypoint  to  the  flight  plan.  If  you  would,  then  press  ACK,  if  not 
then  turn  the  large  left  knob  to  get  out  of  the  FPL  mode, 

3.  Press  ACK  and  the  receiver  will  prompt  you  for  the  waypoint  you  want 
to  select. 

4.  Turn  the  right  outer  knob  to  the  VOR  category.  Enter  the  identifier  PSC 
by  the  same  method  we  used  before,  either  by  scrolling  through  the  list 
or  by  character  entry.  Note:  as  you  are  entering  the  identifier,  the  ACK 
button  will  flash  only  if  you  are  displaying  a  waypoint  that  is  in  the 
database  -  it  will  not  flash  if  there  is  no  waypoint  in  that  category  of  the 
database  with  that  identifier.  For  example,  it  will  flash  if  you  enter  PSC 
VOR  but  not  for  PSD  because  there  is  no  waypoint  in  the  VOR  category 
of  the  database  named  PSD,  If  you  accidentally  press  ACK  when  it  is 
not  flashing,  the  GPS/LORAN  saves  your  present  position  as  a  waypoint. 
demonstrate  Also,  make  sure  that  you  are  in  the  right  category.  If  you 
are  supposed  to  be  entering  a  VOR  waypoint  and  you  are  entering  a 
waypoint  in  the  INT  category  by  accident,  press  the  CRSR  button.  This 
will  not  enter  the  waypoint  but  it  will  allow  you  to  change  the  category 
and  enter  it  in  again. 

5.  When  you  have  PSC  displayed  as  the  desired  waypoint,  press  the 
flashing  ACK  to  enter  it  into  the  flight  plan  and  you  will  be  asked  again  if 
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you  want  to  add  a  waypoint.  DELETE  THIS  WA  YPOINT  FOR  THE 
SUBJECT  NOW. 

Lets  say  we  want  to  enter  a  flight  plan  and  have  JFK  airport  as  our  first 
waypoint  in  this  flight  plan. 

1 .  Turn  the  right  outer  knob  to  APT  and  then  bring  up  JFK. 

2.  Press  the  flashing  ACK  because  it  is  the  waypoint  you  want  and  it  then 
prompts  you  to  add  another  waypoint. 

3.  The  experimenter  will  now  help  you  to  enter  the  rest  of  the  flight  plan. 
The  waypoints  and  their  respective  categories  are  as  follows: 

waypoint  category 

JFK  APT 

ACY  VOR 

NEL  NDB 

CASVI  INT 

FSD  APT 

Show  subject  how  to  check  the  accuracy  of  the  flight  plan  by  scrolling  the  left 
inner  knob  (first  get  out  and  back  into  FPL  mode). 


V.  How  to  Edit  the  Flight  Plan 

During  the  flight  it  may  be  necessary  for  the  pilot  to  edit  the  flight  plan  by 
adding,  deleting  or  changing  waypoints.  Also  if  you  make  a  mistake  while  entering 
the  flight  plan  you  can  correct  it  using  the  editing  functions.  For  example  if  you 
misspell  the  identifier  or  press  the  ACK  button  too  early  you  may  accidentally  enter 
a  wrong  waypoint  into  the  plan.  You  can  use  the  delete  operation  described  below 
to  correct  the  error.  The  basic  procedure  for  the  editing  operations  is  as  follows: 

1 .  Set  the  large  left  knob  to  FPL. 

2.  Turn  the  small  left  knob  to  scroll  to  the  leg  of  the  flight  plan  that  you 
want  to  edit.  If  you  want  to  delete  or  change  a  waypoint,  scroll  the 
flight  plan  so  that  particular  waypoint  is  the  "To"  waypoint.  If  you  want 
to  insert  a  waypoint,  scroll  the  flight  plan  to  display  the  two  waypoints 
between  which  you  want  to  insert  the  new  waypoint. 

3.  Turn  the  large  right  knob  to  the  editing  function  that  you  want. 
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4.  Press  the  flashing  ACK  to  carry  out  the  edit. 

First  lets  change  one  waypoint  to  another.  Change  the  ACY  VOR  waypoint 
to  be  a  new  waypoint. 

1 .  Set  the  left  outer  knob  to  FPL  and  scroll  the  flight  plan  (using  the  left 
inner  knob)  so  "ACY"  is  the  "to"  waypoint. 

2.  Turn  outer  right  knob  to  "Change  ACY  ACK." 

3.  Press  ACK. 

4.  Using  the  knobs  on  the  right,  select  the  category  and  the  specific 
waypoint  to  replace  it,  the  NANCI  intersection. 

5.  Press  the  flashing  ACK  to  enter  it. 

After  demonstrating  this,  allow  them  to  change  a  waypoint. 

Now  we  will  delete  CASVI  INT: 

1 .  With  the  left  outer  knob  set  on  FPL,  scroll  the  flight  plan  so  CASVI  is  the 
"to"  waypoint. 

2.  Turn  the  large  right  knob  to  the  display  message  prompting  you  to  delete 
a  waypoint. 

3.  Press  ACK  to  enter  the  delete.  Allow  them  to  delete  a  waypoint. 

Now  we  will  insert  the  DIXIE  INT  between  the  NANCI  INT  and  the  NEL  NDB: 

1 .  With  the  left  outer  knob  set  on  FPL,  scroll  the  flight  plan  to  display  the 
two  waypoints,  NANCI  and  NEL,.  between  which  you  want  to  insert  the 
new  waypoint,  DIXIE. 

2.  Turn  the  right  outer  knob  to  "Insert  wpt  ACK?" 

3.  Press  ACK. 

4.  Use  the  large  right  knob  to  select  the  category  and  the  small  right  knob 
to  scroll  through  the  waypoints  in  the  flight  plan  or  use  the  character 
entry  method  to  select  the  waypoint  to  insert. 

5.  Press  the  flashing  ACK  to  enter  it. 
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Allow  them  to  insert  a  waypoint. 

You  will  also  notice  there  is  a  prompt  "Add  waypoint  ACK?."  This  is 
acknowledged  if  you  want  to  add  a  waypoint  to  the  end  of  the  flight  plan,  not  to 
insert  a  waypoint  between  two  others.  The  steps  to  add  a  waypoint  in  this  editing 
mode  are  the  same  as  when  you  were  creating  the  flight  plan. 

If  you  have  to  insert  some  waypoints  and  delete  others  in  the  same  segment 
of  the  flight  plan  it  is  easiest  to  use  the  "Change  waypoint"  function.  For  example, 
if  you  have  to  delete  two  waypoints  and  put  in  three  new  ones,  the  easiest  way  is 
to  change  the  first  two  to  new  ones  and  insert  the  other  new  one  rather  than 
deleting  the  three  old  ones  and  inserting  the  new  ones. 
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Day  0  Training 


The  business  day  before,  call  to  remind  the  subject  of  your  appointment.  Be  sure 

to  determine  for  what  test  interval  condition  he/she  will  be  available  (7  or  14).  Tell 

subject  to  bring  corrective  lenses  if  needed. 

Prior  to  subject  arrival: 

1.  Turn  on  simulator  and  other  equipment,  hang  "DO  NOT  DISTURB"  sign,  mark 
video  tapes. 

2.  Assign  proper  subject  number  (1-5),  referring  to  conditions  sheets. 

Pilot  # _ . 

3.  Insert  a  blank,  labelled  diskette  into  drive  b.  Initiate  first  flight  trial  conditions 
by  typing  TUNEGPS1.  SUBJ*  DAY  {@C:\IFT3213).  Verify  turbulence  (1-3)  and 
trial#  and  hit  ENTER,  press  Y  to  continue.  Press  ESC  to  get  rid  of  flight 
instrument  window.  Switch  A/B  box  from  experimenter's  monitor  to  subject's 
monitor, 

4.  Make  certain  all  experimental  materials  are  available  (hard  copies  and  disk), 
including  formatted  diskettes  for  data  files. 

When  subject  arrives: 

Administer  visual  acuity  test  in  full  illumination.  Proceed  upon  successful 

completion.  (If  they  do  not  have  20/20  vision  or  close  to  it,  stop  here). 

Wear  corrective  lenses?  Y  N  Worn  today?  Y  N 

1.  Read  informed  consent  form  and  have  it  signed.  Give  a  charge  number  if 
applicable.  Fill  out  information  sheet  for  check  remittance. 

2.  Administer  pilot  experience  questionnaire. 

3.  Adjust  subject's  seating  relative  to  controls.  Position  the  bar  behind  the 
subject's  chair. 

4.  Turn  lights  down  (left  off;  right  to  black  mark)  and  keep  door  closed  for  the 
remainder  of  the  experiment. 

5.  "You  will  be  asked  to  complete  a  series  of  18  trials  today.  Some  of  these  trials 
will  involve  programming  the  receiver  (entering  and  editing  a  flight  plan),  some 
will  involve  flying  the  simulator,  and  others  will  combine  both  of  these  tasks. 


We  will  begin  by  familiarizing  you  with  the  simulator  and  the  operation  of  the 
receiver."  (Order  depends  on  subject  number). 

6.  If  flight  sessions  are  first,  familiarize  subject  with  simulator  and  flight  task  using 
the  simulator  instructions  and  graphic  presentation. 

7.  Read  description  of  instruments  while  subject  is  viewing  simulator  monitor. 

Give  subject  handout  describing  instruments  for  review. 

Flight  instruments  On  the  display  ooino  clockwise  from  lower  left  corner: 

Automatic  Direction  Finder  (ADF) 

Turn  Coordinator 

The  single  hash  marks  on  the  right  and  left  side  indicate  that  the  aircraft  is  in 
a  "standard  rate"  (3°/second  or  360°/two  minutes)  turn.  Displacement  of 
the  "ball"  indicates  an  uncoordinated  turn  that  can  be  corrected  by  applying 
rudder  on  the  same  side  as  the  ball  (i.e.,  step  on  the  ball). 

Airspeed  Indicator 

The  speed  of  the  aircraft  is  displayed  as  indicated  airspeed  (IAS)  in  knots 
(kts).  The  indicator  has  three  colored  bands  and  a  red  marker.  The  green 
band  is  the  normal  operational  speed  range,  the  white  band  is  the  stall  speed 
range  for  full  flaps,  the  yellow  band  is  a  caution  range,  and  the  red  mark  is 
the  "never  exceed"  speed.  Each  trial  will  always  begin  at  approximately  100 
kts  airspeed  and  will  vary  throughout  a  trial  depending  on  aircraft  attitude. 
DO  NOT  ATTEMPT  TO  ADJUST  AIRSPEED  USING  THE  THROTTLE 
CONTROL. 

Outside  View  Display 

ADF  Tuner  lower  left 

Magnetic  Compass/Marker  Beacons  upper  left 

Navigation  Radio  (NAV)  Cluster  right  side 

Tachometer 

Fuel  Cluster 

Engine  Instruments 

Clock 

Throttle  Cluster 

Elevator/Trim/Flap  Position  Indicators 
(Do  not  attempt  to  adjust  the  trim  and  flaps  during  the  experiment). 

Gyro  Compass/Directional  Gyro 

Provides  more  accurate  heading  information  than  the  traditional  magnetic 
compass.  However,  it  is  vulnerable  to  precession  over  time  and  cannot  be 
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adjusted  to  compensate.  Do  not  use  this  instrument  for  precise  tracking 
information. 

Attitude  Indicator 

Aircraft  pitch  and  bank  are  shown  with  this  instrument.  The  vertical  hash 
marks  are  graduated  in  5®  increments  for  pitch.  The  bank  index  (outer 
circle)  is  graduated  in  10°  increments  up  to  the  first  30®  of  bank,  and  in 
30®  increments  for  up  to  90®  of  bank.  The  blue  portion  of  the  indicator 
shows  the  direction  of  the  sky  and  the  gray  portion  indicates  the  earth 
relative  to  the  aircraft  attitude. 

Altimeter 

Shows  aircraft  altitude  (above  sea  level)  and  is  preset  at  5000  feet  for  each 
trial.  The  barometric  pressure  is  preset  at  29.92  and  does  not  require 
adjustment.  DO  NOT  ATTEMPT  TO  ADJUST  THE  ALTIMETER  DURING  THE 
EXPERIMENT. 

VOR1/VOR2 

The  number  one  VOR  (VI)  is  operational  and  the  course  deviation  needle 
(GDI  needle)  should  be  used  for  primary  tracking  information.  The  GDI 
needle  will  be  displaced  toward  the  left  or  right,  depending  upon  aircraft 
heading,  relative  to  the  tuned-in  course.  DO  NOT  ATTEMPT  TO  ADJUST 
THE  VOR  DURING  THE  EXPERIMENT  TO  COMPENSATE  FOR  COURSE 
DEVIATIONS,  FLY  THE  AIRCRAFT  TO  MAINTAIN  A  CENTERED  CDI 
NEEDLE.  The  second  VOR  (V2)  is  INOPerative  during  the  experiment. 

Vertical  Speed  Indicator 

Displays  the  rate  of  climb  and  descent  in  hundreds  of  feet  per  minute.  This 
instrument  will  vary  according  to  aircraft  attitude. 

Other: 

Rudder  pedals  and  yoke  functional 
Throttle  -  remind  subjects  not  to  use 
No  trim 

Yoke  -  must  be  centered  prior  to  each  flight  trial.  Show  subject  how  to  line 
up  yoke  with  black  line  and  to  use  elevator  instrument  (tick  mark 
between  the  R  and  I  in  TRIM). 

"At  the  end  of  each  flight,  you  will  be  presented  with  two  scores  you'll  use  to 
assess  your  own  performance.  The  first  score  represents  cross-track  error  score 
(XTK),  which  is  computed  as  the  average  horizontal  deviation  away  from  the 
course  line  expressed  in  feet.  Next  you  will  find  your  altitude  score,  which  is 
computed  as  the  average  vertical  deviation  expressed  in  feet.  The  better  your 
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performance  for  either  of  these  measures,  the  lower  your  scores  will  be.  Please  try 
to  keep  your  scores  as  close  to  zero  as  possible." 

8.  Refer  to  conditions  sheet  to  determine  proper  order  for  the  following: 

Some  subjects  fly,  program,  fly  and  program;  other  subjects  program,  fly,  fly 
and  program. 

Flight  Trials 

(FLIGHT  DURATION:  4  MINUTES) 

Instruct  subject  as  follows: 

"Please  maintain  your  altitude  at  +/-100  feet  and  use  the  GDI  to  maintain 
your  course  as  accurately  as  possible.  Before  each  trial  begins,  please  make 
sure  the  yoke  is  centered.  To  verify,  use  the  black  line  on  the  yoke  and  the 
elevator  position  indicator.  Do  you  have  any  questions?  Ready?" 

To  initiate  flight  trials: 

For  trial  1,  press  pause  (P)  to  start;  for  the  remaining  trials... 

1 .  Verify  turbulence  level  and  trial  number,  press  Y  to  continue. 

2.  Press  ESC. 

3.  Switch  A/B  box  from  experimenter's  monitor  to  subject's  monitor. 

4.  Press  pause  (P)  to  start  simulation, 

5.  Say  "OK"  while  depressing  (P)  to  indicate  the  end  of  each  flight. 

6.  Press  9  twice  and  Y,  ENTER. 

7.  Enter  flight  plan  number  for  trial  just  completed. 

8.  Receive  feedback  from  trial. 

9.  Switch  A/B  box  to  experimenter's  screen  after  subject  has  viewed 
feedback. 

10.  Press  Y  to  continue,  begin  next  trial. 

Programming  Trials 

Prepare  the  GPS/LORAN  to  rur  sktop  simulation  mode  and  give  receiver 
operating  instructions  to  the  su  Instruct  the  subject  on  how  to  operate  the 
receiver  verbally  and  with  written  instructions.  Make  sure  to  sit  on  the  right  side 
of  the  subject  while  reviewing  receiver  procedures. 

Place  first  flight  plan  on  knee  board  uoside  down.  Instruct  subject  as  follows: 

"When  you  complete  the  flight  plan,  please  check  it  for  accuracy  and 
verbally  indicate  when  you  are  finished.  Please  erase  the  flight  plan  prior  to 
the  next  trial." 
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Flight/Programming  Trials 

(FLIGHT  DURATION:  SUBJECT  DEPENDENT) 

Be  sure  that  the  GPS/LORAN  is  set  to  FPL  mode  and  does  not  contain  a  flight  plan. 
(?????  should  appear  on  lefthand  side;  "Undefined  Leg "  should  appear  on  the 
right).  Place  first  flight  plan  on  the  knee  board  face  down  and  instruct  the  subject 
as  follows: 

"Please  maintain  your  altitude  at  +/-100  feet  and  use  the  CDI  to  maintain 
your  course  as  accurately  as  possible.  Remember  that  vour  attention  should 
primarily  be  given  to  flying.  Unlike  a  real  aircraft,  if  you  divert  your  attention 
from  flying  for  extended  periods  you  may  experience  difficulty  in  making 
control  adjustments.  I  will  say  begin  after  30  seconds  of  your  flight  have 
elapsed  (USE  STOPWATCH).  At  this  point,  please  turn  the  flight  plan  right- 
side  up  and  begin  programming.  When  you  have  finished  entering  the  flight 
plan,  please  check  it  for  accuracy,  making  sure  to  fix  any  mistakes  that  you 
might  have  made,  and  press  the  PAUSE/PROFILE  button  on  the  control  panel 
(INDICATE).  You  may  erase  the  flight  plan  when  the  trial  is  completed. 
Before  each  trial  begins,  please  make  sure  the  yoke  is  centered.  Do  you 
have  any  questions?  Ready?" 

Initiate  flight  simulation. 

««««Mention  that  yoke  or  pedal  movements  made  in  between  trials  (when 
simulation  is  not  displayed)  can  be  disruptive  to  data  collection  procedures.**** 

9.  Thank  the  subject  for  participating  and  remind  him  to  return  the  next  day  for 
his  first  data  collection  session.  Issue  a  reminder  slip  and  record  times  on 
subject  schedules.  Delete  LASTRIC.PTH.  Backup  (hard  and  floppy)  and  file 
data.  Prepare  summary  of  Subject's  comments. 

Day  1  Data  Collection 

One  business  day  before,  call  to  remind  the  subject  of  your  appointment. 

Prior  to  subject  arrival: 

1.  Turn  on  simulator,  prepare  for  first  flight  trial  and  hang  "DO  NOT  DISTURB" 
sign. 

2.  Make  certain  that  the  GPS/LORAN  is  cleared  of  flight  plans  created  during  the 
latest  session. 

3.  Set  up  camera  and  VCR.  Mark  video  tape. 
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4.  Make  certain  all  experimental  materials  are  available  (hard  copies  and  disk). 
\Nhen  subject  arrives: 

1 .  Worn  corrective  lenses  today?  Y  N  N/A 

2.  Adjust  subject's  seating  relative  to  flight  controls  and  answer  any  questions  the 
subject  might  have  regarding  the  simulator  and  flight  task.  Explain  that  you 
cannot  answer  receiver  operation  questions. 

3.  Turn  lights  down  and  close  the  door  for  the  remainder  of  the  experiment. 

4.  "You  will  be  asked  to  complete  a  series  of  18  trials  today.  Some  of  these  trials 
will  involve  programming  the  receiver,  some  will  involve  flying  the  simulator, 
and  others  will  combine  both  of  these  tasks." 

Flight  Trials 

(FLIGHT  DURATION;  4  MINUTES) 

Instruct  subject  as  follows: 

"Please  maintain  your  altitude  at  +/-100  feet  and  use  the  GDI  to  maintain 
your  course  as  accurately  as  possible.  Before  each  trial  begins,  please  make 
sure  the  yoke  is  centered." 

Initiate  flight  simulation.  Repeat  procedure  for  remaining  flights.  Note  that 
crashed  trials  should  be  repeated. 

Programming  Trials  (remember  to  video  record) 

Be  sure  that  the  GPS/LORAN  is  set  to  FPL  mode  and  does  not  contain  a  flight  plan 
(?????  should  appear  on  lefthand  side;  "Undefined  Leg"  should  appear  on  the 
right). 

Place  first  flight  plan  on  knee  board  upside  down.  Instruct  subject  as  follows: 

"When  you  complete  the  flight  plan,  please  check  it  for  accuracy,  making 
sure  to  fix  any  mistakes  that  you  might  have  made,  and  verbally  indicate 
when  you  are  finished.  Please  erase  the  flight  plan  prior  to  the  next  trial." 

Flight/Programming  Trials 

Be  sure  that  the  GPS/LORAN  is  set  to  FPL  mode  and  does  not  contain  a  flight  plan. 
Place  first  flight  plan  on  knee  board  and  instruct  the  subject  as  follows: 

"Please  maintain  your  altitude  at  +/-100  feet  and  use  the  GDI  to  maintain 
your  course  as  accurately  as  possible.  Remember  that  vour  attention  should 
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primarily  be  given  to  f lying.  I  will  say  begin  after  30  seconds  of  your  flight 
have  elapsed  (USE  STOPWATCH).  At  this  point,  please  turn  the  flight  plan 
right-side  up  and  begin  programming.  When  you  have  finished  entering  the 
flight  plan,  please  check  it  for  accuracy,  making  sure  to  fix  any  mistakes 
that  you  might  have  made,  and  press  the  PAUSE/PROFILE  button  on  the 
control  panel  (INDICATE).  You  may  erase  the  flight  plan  when  the  trial  is 
completed.  Before  each  trial  begins,  please  make  sure  the  yoke  is  centered. 
Do  you  have  any  questions?  Ready?" 

Initiate  flight  simulation.  Repeat  for  remaining  flights,  recording  programming 
errors  (video  and  manual)  for  each  flight  trial. 

After  each  trial,  ask  the  subject  reasons  for  error  commission  (use  the  checklist). 
Record  the  point  at  which  the  subject  starts  programming  during  flight  and  the 
time  the  subject  finishes  the  last  control  manipulation  for  each  flight. 

Note:  Use  the  stopwatch  for  recording  F  trials  (4  minutes).  Use  clock  on  video 
monitor  to  record  time  for  P  and  FP  trials. 


begin 

end 

Tot 

Trial  1 

F  P  FP 

begin 

end 

Tot 

Trial  2 

F  P  FP 

begin 

end 

Tot 

Trial  3 

F  P  FP 

begin 

end 

Tot 

Trial  4 

F  P  FP 

begin 

end 

Tot 

Trial  5 

F  P  FP 

begin 

end 

Tot 

Trial  6 

F  P  FP 

begin 

end 

Tot 

Trial  7 

F  P  FP 

begin 

end 

Tot 

Trial  8 

F  P  FP 

begin 

end 

Tot 

Trial  9 

F  P  FP 

begin 

end 

Tot 

Trial  10 

F  P  FP 

begin 

end 

Tot 

Trial  1 1 

F  P  FP 

begin 

end 

Tot 

Trial  12 

F  P  FP 

begin 

end 

Tot 

Trial  13 

F  P  FP 

begin 

end 

Tot 

Trial  14 

F  P  FP 

begin 

end 

Tot 

Trial  15 

F  P  FP 

begin 

end 

Tot 

Trial  16 

F  P  FP 

begin 

end 

Tot 

Trial  17 

F  P  FP 

begin 

end 

Tot 

Trial  18 

F  P  FP 

Delete  LASTRIC.PTH.  Thank  subject  for  participating  and  remind  subject  to  return 
in  either  7  or  14  days  (check  the  proper  condition).  Issue  a  reminder  slip  and 
record  times  on  subject  schedules.  File  and  download  all  data. 
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One  business  day  before,  call  to  remind  the  subject  of  your  appointment. 

Prior  to  subject  arrival: 

1.  Turn  on  simulator,  prepare  for  first  flight  trial,  and  hang  "DO  NOT  DISTURB" 
sign. 

2.  Make  certain  that  the  GPS/LORAN  is  cleared  of  flight  plans  created  during  the 
latest  session. 

3.  Set  up  camera  and  VCR. 

4.  Make  certain  all  experimental  materials  are  available  (nard  copies  and  disk). 

When  subject  arrives: 

1 .  Worn  corrective  lenses  today?  Y  N  N/A 

2.  Adjust  subject's  seating  relative  to  flight  controls  and  answer  any  questions  the 
subject  may  have  regarding  the  simulator  and  flight  task.  Explain  that  you 
cannot  answer  receiver  operation  questions. 

3.  Turn  lights  down  and  close  the  door  for  the  remainder  of  the  experiment. 

4.  "You  will  be  asked  to  complete  a  series  of  14  trials  today.  Some  of  these  trials 
will  involve  flying  the  simulator,  and  others  will  combine  flying  and  the  receiver 
programming  task." 

Flight  Trials 

(FLIGHT  DURATION:  4  MINUTES) 

Instruct  subject  as  follows: 

"Please  maintain  your  altitude  at  +/-100  feet  and  use  the  GDI  to  maintain 
your  course  as  accurately  as  possible.  Before  each  trial  begins,  please  make 
sure  the  yoke  is  centered." 

Initiate  flight  simulation.  Use  stopwatch  to  monitor  30  second  mark  and  tell 

subject  to  begin.  Repeat  procedure  for  remaining  flights.  Note  that  crashed  trials 

should  be  repeated. 


Flight/Programming  Trials 

Be  sure  that  the  GPS/LORAN  is  set  to  FPL  mode  and  does  not  contain  a  flight  plan. 
Place  first  flight  plan  on  knee  board  and  instruct  the  subject  as  follows: 

"Please  maintain  your  altitude  at  +/-100  feet  and  use  the  GDI  to  maintain 
your  course  as  accurately  as  possible.  Remember  that  vour  attention  should 
primarily  be  given  to  flvino.  I  will  say  begin  after  30  seconds  of  your  flight 
have  elapsed  (USE  STOPWATCH).  At  this  point,  please  turn  the  flight  plan 
right-side  up  and  begin  programming.  When  you  have  finished  entering  the 
flight  plan,  please  check  it  for  accuracy,  making  sure  to  fix  any  mistakes 
that  you  might  have  made,  and  press  the  PAUSE/PROFILE  button  on  the 
control  panel  (INDICATE).  You  may  erase  the  flight  plan  when  the  trial  is 
completed.  Before  each  trial  begins,  please  make  sure  the  yoke  is  centered. 
Any  questions?  Ready?" 

Initiate  flight  simulation.  Repeat  for  remaining  flights,  recording  oroorammino 
errors  (video  and  manual)  for  each  flight  trial. 

After  each  trial,  ask  the  subject  reasons  for  error  commission  (use  the  checklist). 
Record  the  point  at  which  the  subject  starts  programming  during  flight  and  the 
time  the  subject  finishes  the  last  control  manipulation  for  each  flight.  Continually 
monitor  subject's  programming  manipulations  and  check  against  flight  plans. 

Note:  Use  stopwatch  for  recording  F  trials  (4  minutes).  Use  clock  on  video 
monitor  to  record  time  for  P  and  FP  trials. 


begin 

end 

Tot 

Trial  1 

F  P  FP 

begin 

end 

Tot 

Trial  2 

F  P  FP 

begin 

end 

Tot 

Trial  3 

F  P  FP 

begin 

end 

Tot 

Trial  4 

F  P  FP 

begin 

end 

Tot 

Trial  5 

F  P  FP 

begin 

end 

Tot 

Trial  6 

F  P  FP 

begin 

end 

Tot 

Trial  7 

F  P  FP 

begin 

end 

Tot 

Trial  8 

F  P  FP 

begin 

end 

Tot 

Trial  9 

F  P  FP 

begin 

end 

Tot 

Trial  10 

F  P  FP 

begin 

end 

Tot 

Trial  1 1 

F  P  FP 

begin 

end 

Tot 

Trial  12 

F  P  FP 

begin 

end 

Tot 

Trial  13 

F  P  FP 

begin 

end 

Tot 

Trial  14 

F  P  FP 
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5.  Administer  post-experimeni  questionnaire. 

6.  Thank  subject  for  participating. 

7.  Delete  LASTRIC.PTH 

8.  Backup  and  file  all  data  and  manually  record  errors. 
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INFORMED  CONSENT  FORM 
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INFORMED  CONSENT  FORM 


You  have  been  asked  to  participate  in  the  Receiver  Programming  Performance 
Study  conducted  by  the  Operator  Performance  and  Safety  Analysis  Division  of  the 
Voipe  Center. 

The  purposes  of  this  study  are  to  determine  how  long  a  person  retains  the 
knowledge  required  to  operate  a  GPS/LORAN  receiver,  to  identify  the  GPS/LORAN 
operations  that  are  most  prone  to  error,  and  to  identify  the  types  of  errors  that  are 
made  while  performing  GPS/LORAN  operations.  If  you  agree  to  participate,  you 
will  be  asked  to  complete  a  brief  questionnaire  regarding  your  flight  experience  and 
any  previous  experience  you  may  have  had  with  using  GPS/LORAN  systems.  You 
will  then  be  familiarized  with  a  flight  simulator  and  trained  to  operate  the  receiver. 
Additionally,  you  will  be  asked  to  perform  various  programming  functions  using  the 
GPS/LORAN  system  during  flight  simulation. 

The  experiment  will  take  up  to  10  hours  to  complete  over  a  period  of  three  days. 
There  are  no  risks  involved  in  participating  in  this  study.  Please  do  not  hesitate  to 
ask  questions  about  the  study  at  any  time.  Your  data  will  be  kept  strictly 
confidential  and  your  name  will  not  be  associated  with  your  data. 

Your  participation  in  this  study  is  strictly  voluntary.  If  you  agree  to  participate,  you 
will  either  be  given  a  Voipe  Center  account  number  to  which  you  may  charge  your 
time,  or  you  will  be  paid  $100.00.  You  are  free  to  withdraw  at  any  time  without 
penalty.  Your  cooperation  is  sincerely  appreciated. 


Vicki  Shearer  and  Ed  Madigan 
Researchers 

Battelie  Memorial  Institute 

Voipe  National  Transportation  Systems  Center 

(617)  494-2464  and  (614)  424-3325 


Signature  and  Age  of  Participant 


Date 


Name  (please  print) 


Address  and  Phone  Number 


Signature  of  researcher 


Date 
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APPENDIX  F 


PILOT  EXPERIENCE  QUESTIONNAIRE 
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Pilot  Experience  Questionnaire 


1 .  Age: _ 

2.  Gender:  Male  Female 

3.  Please  estimate  your  flight  times  as  accurately  as  you  can: 

total  flight  time _ hours 

total  IFR  flight  time _ hours 

4.  Please  indicate  the  type  of  aviation  experience  you  have.  Check  all  that  apply. 

Part  121  _ General  Aviation _ 

Part  1 35 _ Corporate  _ 

Military _ 

Please  list  the  type(s)  of  aircraft  currently  flown: 


5.  Have  you  ever  used  a  GPS/LORAN  receiver  prior  to  this  evaluation? 

YES  NO 

IF  YES,  please  answer  the  remaining  questions. 

6.  Please  list  all  receivers  you  currently  use: _ 


Other  receivers  used: 


7.  Please  estimate  the  percentage  of  time  you  use  your  receiver  during  flights: 
% 
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8.  Does  operating  the  GPS/LORAN  receiver  distract  you  from  your  primary  flying 
tasks? 

YES  NO 

If  YES.  how? 

9.  Would  you  recommend  any  design  changes?  YES  NO  If  YES,  please 

explain. 

Knobs/Keys: 

Labeling: 

Displays: 

Procedures: 


For  the  receiver  you  use  the  most  (please  specify  _ ),  answer  the 

following; 

10.  Using  the  scale  below,  please  rate  the  level  of  ease/difficulty  associated  with 
performing  each  of  the  following  functions. 


VERY 

VERY 

Functions 

EASY 

NEUTRAL 

DIFFICULT 

Reading  the  display  panel  1 

2 

3 

4 

5 

6 

7 

Creating  a  flight  plan 

1 

2 

3 

4 

5 

6 

7 

Changing  a  flight  plan 

1 

2 

3 

4 

5 

6 

7 

Overall  usability 

1 

2 

3 

4 

5 

6 

7 

Recovery  from  errors 

1 

2 

3 

4 

5 

6 

7 

Operating  the  control 

1 

2 

3 

4 

5 

6 

7 

APPENDIX  G 


POST-EXPERIMENT  QUESTIONNAIRE 
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1 .  After  using  the  GPS/LORAN  receiver,  which  of  the  programming  procedures  were 
most  confusing  to  you?  (i.e.,  recatling  waypoints,  creating  flight  plans,  editing 
flight  plans)  Why? 


2.  In  your  opinion,  what  is  the  single  best  feature  of  the  GPS/LORAN?  Why? 


3.  In  your  opinion,  what  is  the  single  worst  feature  of  the  GPS/LORAN?  Why? 


4.  Did  you  have  any  difficulties  with  retrieving  information? 

YES  NO 


If  yes,  please  explain. 


5.  Did  you  find  that  the  receiver  programming  logic  corresponds  with  the  way  you 
think  about  following  a  flight  plan? 

YES  NO 
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6.  If  you  use  GPS/LORAN  when  you  fly,  do  you  usually  follow  a  flight  plan  or  fly 
waypoint  to  waypoint?  Please  circle  one  response. 

a.  Not  applicable  -I  do  not  use  GPS/LORAN  when  I  fly 

b.  I  follow  a  flight  plan 

c.  I  fly  waypoint  to  waypoint 

7.  What  types  of  navigation  information  would  you  prefer  to  see  displayed  on  the 
GPS/LORAN  monitor  at  all  times?  Please  rank  your  choices  in  order  of  importance 
and  provide  an  explanation. 


8.  Would  you  recommend  any  design  changes  to  the  receiver  you  used  for  this 
experiment? 


YES  NO  If  YES,  please  explain. 


Knobs/Keys: 


Labeling; 


Displays: 


Procedures: 
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POST-EXPERIMENT  QUESTIONNAIRE  RESPONSES 
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POST-EXPERIMENT  QUESTIONNAIRE  RESPONSES 

Responses  are  given  in  bold,  the  response  frequency  is  given  to  the  left. 

1 .  After  using  the  GPS/LORAN  receiver,  which  of  the  programming  procedures  were 
most  confusing  to  you?  (/.e.,  recalling  waypoints,  creating  flight  plans,  editing 
flight  plans)  Why? 

(1)  The  procedures  themselves  were  explained  well,  however  the  L.E.D.  display 
could  be  laid  out  better. 

(1)  Editing.  Seemed  to  be  easier  to  misidentify  the  type  of  waypoint. 

(3)  No  confusion 

2.  In  your  opinion,  what  is  the  single  best  feature  of  the  GPS/LORAN?  Why? 

(1)  "Direct  To."  Very  useable  enroute.  Position  Accuracy. 

(1)  Being  able  to  fly  direct  to  any  point  at  all  including  intersections  and  airports. 
(1)  Direct  navigation:  faster. 

(1)  Bright  display. 

(1)  Nothing  in  particular  stood  out. 

3.  In  your  opinion,  what  is  the  single  worst  feature  of  the  GPS/LORAN?  Why? 

(1)  Nothing  in  particular  stood  out. 

(1)  In  the  operation  instructions,  use  of  both  inner  and  outer  and  large  and  small 
to  describe  knobs.  Preferred  large  and  small. 

(1)  Nav.  arror;  i.e.  poor  triangulation,  especially  in  precip. 

(1)  Although  I  have  not  experienced  it.  I've  heard  that  reception  can  sometimes 
be  disrupted  especially  when  needed  most;  In  hard  IFR  with  heavy  rain  or 
snow. 

(1)  Time  to  scan  a  waypoint. 
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4.  Did  you  have  any  difficulties  with  retrieving  information?  If  yes,  please  explain. 

(3)  NO 

(1)  YES;  Display  needs  to  be  larger  and  better  spaced  for  easier  identification. 

(1)  YES;  Several  waypoints,  as  given,  were  not  in  database.  I  had  to  substitute 
something  "on  the  fly." 

5.  Did  you  find  that  the  receiver  programming  logic  corresponds  with  the  way  you 
think  about  following  a  flight  plan? 

(4)  YES 

(1)  No  way  to  answer  Y  or  N  based  on  this  experiment,  since  there  is  no 
requirement  to  actually  follow  GPS/LORAN  course  guidance. 

6.  If  you  use  GPS/LORAN  when  you  fly,  do  you  usually  follow  a  flight  plan  or  fly 
waypoint  to  waypoint?  Please  circle  one  response. 

(2)  a.  Not  applicable  -I  do  not  use  GPS/LORAN  when  I  fly 

(1)  b.  I  follow  a  flight  plan 

(2)  c.  I  fly  waypoint  to  waypoint 

7.  What  types  of  navigation  information  would  you  prefer  to  see  displayed  on  the 
GPS/LORAN  monitor  at  all  times?  Please  rank  your  choices  in  order  of  importance 
and  provide  an  explanation. 

(1)  If  I  were  using  one,  I  would  like:  1)  Course  to  fly,  2)  Course  deviation,  3) 
Distance  to  waypoint,  4)  Time  to  waypoint 

(1)  1)  Waypoint  I  am  flying  direct  to  as  directed  by  CDI,  2)  Distance  to  the 
waypoint  I  am  flying  to  with  approx  eta  according  to  current  ground  speed 

(1)  1)  Good  waypoint  alert  info.,  2)  Time  to  fix  (minimize  fix  overrun) 

(1)  1)  That  it  is  working,  2)  Crosstrack  error  in  digital  form  i.e.  tenths  of  a  mile 
and  which  direction  to  correct,  3)  Waypoint  name,  4)  Distance/Time  to 
waypoint 

(1)  1)  Speed,  2)  Waypt.  to  which  I'm  navigating,  3)  X-track  error  in  terms  of  CDI 
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8.  Would  you  recommend  any  design  changes  to  the  receiver  you  used  for  this 
experiment?  If  YES,  please  explain. 

(2)  YES 

(3)  NO 

Knobs/Keys: 

( 1 )  Make  keys  slightly  larger  and  beep  when  pressed 

(1)  Should  be  able  to  slow  to  a  letter,  i.e,  faster  you  turn,  the  more  letters  are 
skipped 

Labeling: 

(1)  Poor;  only  selected  item  is  lit.  Very  hard  to  see  in  dim  light  of  test  room 
(1)  Slightly  larger 

Displays: 

(1)  Dots  tend  to  smear-maybe  due  to  my  aging  eyes.  Would  prefer  LCD 
segments  or  gas  discharge 

(1)  Slightly  larger  and  space  between  5  letter  ID  and  the  word  ACK 

Procedures: 

(1)  ok 
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GENERAL  COMMENTS 
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GENERAL  COMMENTS 


(T  refers  to  training  session,  1  refers  to  Test  Day  1,  2  refers  to  Test  Day  2) 

GPS/LORAN  COMMENTS 

Day  Comment 

T  Display  could  be  larger  -  it's  ok  sitting  still  but  in  turbulence  a  bust! 
Numbers  too  small  unless  right  in  front  of  you... 

T  Buttons  too  small  for  fingers  and  too  hard  to  press... 

T  Confusing  which  functions  to  perform  on  which  side  (primary  v.  secondary 
display)... that's  why  I  never  liked  this  unit 

T  There  are  many  ways  to  perform  the  same  function  correctly  can  get 
confusing  to  remember  an  exact  procedure 

T  Scrolling  appears  to  work  differently  on  the  primary  side  than  on  the 
secondary  side  (must  first  hit  cursor  on  secondary  side) 

1  The  waypoint  CDI#1  is  confusing  since  only  3  letters  for  an  NDB  is 
expected 

1  Add  space  between  ACK?  and  Waypoint  identifier... confusing  at  times 

1  The  waypoint  CDI#1  confusing,  has  1  character  more  than  usual 

SIMULATOR/APPARATUS  COMMENTS 

Day  Comment 

T  Feels  like  it's  flying  an  engine  out 

T  Feels  like  the  yoke  is  off  -  won't  go  straight 

T  Simulation  is  sensitive  and  pitch  is  strange 

T  Tendency  to  turn  left  as  if  a  heavy  left  wing 

T  Roll  unstable  at  this  point  or  I'm  not  very  adept  (high  turbulence  condition) 
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GENERAL  COMMENTS 


SIMULATOR/APPARATUS  COMMENTS  (continued) 

Day  Comment 

T  The  angle  of  the  GPS/LORAN  is  disturbing  (physical  fatigue) 

T  The  simulation  is  somewhat  unresponsive 

1  Today  the  simulation  feels  closer  to  full-scale  real 

1  Some  shoulder  discomfort  due  to  position  of  GPS/LORAN  receiver 

1  Responding  (simulator)  slow,  feels  like  runaway  trim  in  a  plane... constantly 
fighting  to  hold  it  straight... 

2  Harder  to  keep  nose  straight  and  level  -  acts  like  severe  turbulence  at 
times... 


EXPERIMENTAL  TASK  COMMENTS 

Day  Comment 

T  Simulation  does  not  respond  like  a  real  aircraft... 

Unable  to  make  1  brisk  control  input  to  correct  course  when  attention  is 
turned  back  to  flying  (from  programming). 

Required  strategy  is  to  program  in  a  series  of  sub-tasks  rather  than  in  one 
large  chunk 

T  It  is  difficult  to  program  and  fly,  the  task  is  demanding  and  it  is  hard  to  split 
attention 

T  I'm  concerned  how  to  fly  and  program  together,  usually  I  turn  on  the 
autopilot 

T  Have  to  know  procedures  down  pat  -  no  time  to  hover  and  think  -  just  do 

2  For  NDB's  and  INT's  I  use  CRSR  entry,  for  other  categories  I  scroll  for  the 

correct  waypoint  (because  there  are  fewer  waypoints  in  the  other 
categories) 
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